[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:43:43 dist-test-slave-n6xs krb5kdc[395](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:43:43 dist-test-slave-n6xs krb5kdc[395](info): set up 2 sockets
Sep 28 08:43:43 dist-test-slave-n6xs krb5kdc[395](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 08:43:44.592958 388 mini_kdc.cc:121] Time spent starting KDC: real 2.062s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260928 08:43:44.772110 388 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.179s user 0.001s sys 0.004s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:43:44 dist-test-slave-n6xs krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:43:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:43:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:43:44.921993 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34887
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34887
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:43:45.251389 408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:43:45.251750 408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:43:45.251976 408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:43:45.259207 408 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:43:45.259763 408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:43:45.259902 408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:43:45.260015 408 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:43:45.260092 408 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:43:45.270174 408 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:43:45.271566 408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:43:45.273011 408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:43:45.285673 416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:45.285744 414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:45.285717 413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:43:45.286494 408 server_base.cc:1034] running on GCE node
I20260928 08:43:45.287343 408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:43:45.288905 408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:43:45.290166 408 hybrid_clock.cc:648] HybridClock initialized: now 1790585025290086 us; error 89 us; skew 500 ppm
I20260928 08:43:45.290566 408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:43:45 dist-test-slave-n6xs krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:43:45.293797 408 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:43:45.295394 408 webserver.cc:460] Webserver started at http://127.0.97.62:45197/ using document root <none> and password file <none>
I20260928 08:43:45.296192 408 fs_manager.cc:359] Metadata directory not provided
I20260928 08:43:45.296293 408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:43:45.296548 408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:43:45.299566 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "c5cabb292ae34c8cb4732d8641a47928"
format_stamp: "Formatted at 2026-09-28 08:43:45 on dist-test-slave-n6xs"
server_key: "f9dde602e330a7334955aa26af78ac80"
server_key_iv: "5eb6b2617830d687f4f5be842206bcb9"
server_key_version: "encryptionkey@0"
I20260928 08:43:45.300415 408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "c5cabb292ae34c8cb4732d8641a47928"
format_stamp: "Formatted at 2026-09-28 08:43:45 on dist-test-slave-n6xs"
server_key: "f9dde602e330a7334955aa26af78ac80"
server_key_iv: "5eb6b2617830d687f4f5be842206bcb9"
server_key_version: "encryptionkey@0"
I20260928 08:43:45.306526 408 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260928 08:43:45.310285 423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:43:45.311734 408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 08:43:45.312004 408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "c5cabb292ae34c8cb4732d8641a47928"
format_stamp: "Formatted at 2026-09-28 08:43:45 on dist-test-slave-n6xs"
server_key: "f9dde602e330a7334955aa26af78ac80"
server_key_iv: "5eb6b2617830d687f4f5be842206bcb9"
server_key_version: "encryptionkey@0"
I20260928 08:43:45.312181 408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:43:45.342168 408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:43:45.346835 408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:43:45.347087 408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:43:45.366050 408 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34887
I20260928 08:43:45.366079 474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34887 every 8 connection(s)
I20260928 08:43:45.367479 408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:43:45.369567 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 408
I20260928 08:43:45.369835 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:43:45.371372 475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:43:45.372244 388 external_mini_cluster.cc:1428] Setting key d3f7cc28c91a8d19637f800c855286aa
I20260928 08:43:45.381701 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5cabb292ae34c8cb4732d8641a47928: Bootstrap starting.
I20260928 08:43:45.386278 475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5cabb292ae34c8cb4732d8641a47928: Neither blocks nor log segments found. Creating new log.
I20260928 08:43:45.387655 475 log.cc:824] T 00000000000000000000000000000000 P c5cabb292ae34c8cb4732d8641a47928: Log is configured to *not* fsync() on all Append() calls
I20260928 08:43:45.391312 475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5cabb292ae34c8cb4732d8641a47928: No bootstrap required, opened a new log
I20260928 08:43:45.403301 475 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5cabb292ae34c8cb4732d8641a47928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5cabb292ae34c8cb4732d8641a47928" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34887 } }
I20260928 08:43:45.403682 475 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5cabb292ae34c8cb4732d8641a47928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:43:45.403868 475 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5cabb292ae34c8cb4732d8641a47928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5cabb292ae34c8cb4732d8641a47928, State: Initialized, Role: FOLLOWER
I20260928 08:43:45.404479 475 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5cabb292ae34c8cb4732d8641a47928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5cabb292ae34c8cb4732d8641a47928" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34887 } }
I20260928 08:43:45.404767 475 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5cabb292ae34c8cb4732d8641a47928 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:43:45.404911 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5cabb292ae34c8cb4732d8641a47928 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:43:45.405102 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5cabb292ae34c8cb4732d8641a47928 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:43:45.408119 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5cabb292ae34c8cb4732d8641a47928 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5cabb292ae34c8cb4732d8641a47928" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34887 } }
I20260928 08:43:45.408751 475 leader_election.cc:302] T 00000000000000000000000000000000 P c5cabb292ae34c8cb4732d8641a47928 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5cabb292ae34c8cb4732d8641a47928; no voters:
I20260928 08:43:45.409417 475 leader_election.cc:288] T 00000000000000000000000000000000 P c5cabb292ae34c8cb4732d8641a47928 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:43:45.409533 480 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5cabb292ae34c8cb4732d8641a47928 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:43:45.411441 480 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5cabb292ae34c8cb4732d8641a47928 [term 1 LEADER]: Becoming Leader. State: Replica: c5cabb292ae34c8cb4732d8641a47928, State: Running, Role: LEADER
I20260928 08:43:45.412159 475 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5cabb292ae34c8cb4732d8641a47928 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:43:45.412042 480 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5cabb292ae34c8cb4732d8641a47928 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5cabb292ae34c8cb4732d8641a47928" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34887 } }
I20260928 08:43:45.419256 481 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5cabb292ae34c8cb4732d8641a47928 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5cabb292ae34c8cb4732d8641a47928" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5cabb292ae34c8cb4732d8641a47928" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34887 } } }
I20260928 08:43:45.419724 481 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5cabb292ae34c8cb4732d8641a47928 [sys.catalog]: This master's current role is: LEADER
I20260928 08:43:45.419312 482 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5cabb292ae34c8cb4732d8641a47928 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5cabb292ae34c8cb4732d8641a47928. Latest consensus state: current_term: 1 leader_uuid: "c5cabb292ae34c8cb4732d8641a47928" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5cabb292ae34c8cb4732d8641a47928" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34887 } } }
Sep 28 08:43:45 dist-test-slave-n6xs krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:43:45.420488 482 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5cabb292ae34c8cb4732d8641a47928 [sys.catalog]: This master's current role is: LEADER
I20260928 08:43:45.421185 486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:43:45.427884 486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:43:45.437055 477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:45.376054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42068 (local address 127.0.97.62:34887)
0928 08:43:45.377059 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:45.377074 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:45.383487 (+ 6413us) server_negotiation.cc:407] Connection header received
0928 08:43:45.397407 (+ 13920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:45.397436 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:45.397917 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:45.398714 (+ 797us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:45.402906 (+ 4192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:45.404530 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:45.407302 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:45.408174 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:45.408434 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:45.423457 (+ 15023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:45.423481 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:45.423505 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:45.423564 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:45.428707 (+ 5143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:45.430017 (+ 1310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:45.430027 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:45.430035 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:45.430256 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:45.430856 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:45.430862 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:45.430865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:45.431657 (+ 792us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:45.434795 (+ 3138us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:45.435103 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 08:43:45.435559 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":517,"threads_started":1}
I20260928 08:43:45.445590 486 catalog_manager.cc:1347] Generated new cluster ID: 1ff74e63b2c74832af3b87dcf63028fa
I20260928 08:43:45.445727 486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:43:45.496191 486 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:43:45.498102 486 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:43:45.508355 486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5cabb292ae34c8cb4732d8641a47928: Generated new TSK 0
I20260928 08:43:45.509238 486 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:43:45.560495 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34887
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:43:45.921413 501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:43:45.921804 501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:43:45.922000 501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:43:45.929852 501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:43:45.930055 501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:43:45.930243 501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:43:45.941144 501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34887
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:43:45.942612 501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:43:45.944360 501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:43:45.958901 506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:45.959215 509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:45.960007 507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:43:45.960381 501 server_base.cc:1034] running on GCE node
I20260928 08:43:45.961092 501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:43:45.962198 501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:43:45.963507 501 hybrid_clock.cc:648] HybridClock initialized: now 1790585025963428 us; error 55 us; skew 500 ppm
I20260928 08:43:45.963999 501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:43:45 dist-test-slave-n6xs krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:43:45.967458 501 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:43:45.969241 501 webserver.cc:460] Webserver started at http://127.0.97.1:32991/ using document root <none> and password file <none>
I20260928 08:43:45.970033 501 fs_manager.cc:359] Metadata directory not provided
I20260928 08:43:45.970178 501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:43:45.970516 501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:43:45.973752 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-0/data/instance:
uuid: "2e42bb3e16424ac19718ce33e6fe1f2e"
format_stamp: "Formatted at 2026-09-28 08:43:45 on dist-test-slave-n6xs"
server_key: "35a030bb024dbdb8f6dff2daff8ed52a"
server_key_iv: "12c9517b2fa96462905120d7bf2eb7ff"
server_key_version: "encryptionkey@0"
I20260928 08:43:45.974644 501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2e42bb3e16424ac19718ce33e6fe1f2e"
format_stamp: "Formatted at 2026-09-28 08:43:45 on dist-test-slave-n6xs"
server_key: "35a030bb024dbdb8f6dff2daff8ed52a"
server_key_iv: "12c9517b2fa96462905120d7bf2eb7ff"
server_key_version: "encryptionkey@0"
I20260928 08:43:45.980473 501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 08:43:45.984349 516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:43:45.986275 501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:43:45.986495 501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-0/wal
uuid: "2e42bb3e16424ac19718ce33e6fe1f2e"
format_stamp: "Formatted at 2026-09-28 08:43:45 on dist-test-slave-n6xs"
server_key: "35a030bb024dbdb8f6dff2daff8ed52a"
server_key_iv: "12c9517b2fa96462905120d7bf2eb7ff"
server_key_version: "encryptionkey@0"
I20260928 08:43:45.986720 501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:43:46.009933 501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:43:46.014693 501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:43:46.014993 501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:43:46.016137 501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:43:46.017611 501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:43:46.017688 501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:43:46.017848 501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:43:46.017910 501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:43:46.055359 501 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35207
I20260928 08:43:46.055415 628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35207 every 8 connection(s)
I20260928 08:43:46.056797 501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:43:46.060263 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 501
I20260928 08:43:46.060487 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:43:46.061071 388 external_mini_cluster.cc:1428] Setting key 1f8a1a9128679792dcf5d8f0d5a4ff00
Sep 28 08:43:46 dist-test-slave-n6xs krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 08:43:46.078429 631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:46.059999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53335 (local address 127.0.97.62:34887)
0928 08:43:46.060472 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:46.060480 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:46.061689 (+ 1209us) server_negotiation.cc:407] Connection header received
0928 08:43:46.063844 (+ 2155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:46.063850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:46.063945 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:46.064077 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:46.065631 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:46.066378 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:46.068013 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:46.068432 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:46.068576 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:46.072492 (+ 3916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:46.072498 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:46.072501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:46.072534 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:46.075142 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:46.076000 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:46.076004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:46.076007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:46.076117 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:46.076592 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:46.076597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:46.076600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:46.076950 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:46.077113 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:46.077826 (+ 713us) server_negotiation.cc:299] Negotiation successful
0928 08:43:46.078054 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":223,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260928 08:43:46.079494 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:46.060648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34887 (local address 127.0.97.1:53335)
0928 08:43:46.061482 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:46.061528 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:46.063516 (+ 1988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:46.064339 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:46.064348 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:46.064877 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:43:46.065460 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:46.065478 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:46.066553 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:46.066560 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:46.067860 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:46.067877 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:46.068664 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:46.068673 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:46.068918 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:46.069682 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:43:46.069705 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:43:46.072240 (+ 2535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:43:46.075332 (+ 3092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:46.075342 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:46.075364 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:46.075849 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:46.076236 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:46.076240 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:46.076245 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:46.076471 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:46.077080 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:43:46.077088 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:43:46.077457 (+ 369us) client_negotiation.cc:769] Sending connection context
0928 08:43:46.077734 (+ 277us) client_negotiation.cc:240] Negotiation successful
0928 08:43:46.078287 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":349,"threads_started":1}
I20260928 08:43:46.081239 629 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34887
I20260928 08:43:46.081619 629 heartbeater.cc:464] Registering TS with master...
I20260928 08:43:46.082921 629 heartbeater.cc:511] Master 127.0.97.62:34887 requested a full tablet report, sending...
I20260928 08:43:46.085269 440 ts_manager.cc:194] Registered new tserver with Master: 2e42bb3e16424ac19718ce33e6fe1f2e (127.0.97.1:35207)
I20260928 08:43:46.086727 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53335
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 08:43:46.101423 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34887
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:43:46.447454 635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:43:46.447767 635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:43:46.447947 635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:43:46.455648 635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:43:46.455785 635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:43:46.455996 635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:43:46.465885 635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34887
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:43:46.467223 635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:43:46.468540 635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:43:46.482412 640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:46.482323 641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:46.482298 643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:43:46.483721 635 server_base.cc:1034] running on GCE node
I20260928 08:43:46.484386 635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:43:46.485353 635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:43:46.486611 635 hybrid_clock.cc:648] HybridClock initialized: now 1790585026486553 us; error 45 us; skew 500 ppm
I20260928 08:43:46.487010 635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:43:46 dist-test-slave-n6xs krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:43:46.490242 635 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:43:46.491756 635 webserver.cc:460] Webserver started at http://127.0.97.2:44705/ using document root <none> and password file <none>
I20260928 08:43:46.492506 635 fs_manager.cc:359] Metadata directory not provided
I20260928 08:43:46.492606 635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:43:46.492856 635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:43:46.496263 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-1/data/instance:
uuid: "d0c974123bb74df18593b275e93225d4"
format_stamp: "Formatted at 2026-09-28 08:43:46 on dist-test-slave-n6xs"
server_key: "c8b0af2ba13daa5e89519a01f3bec4aa"
server_key_iv: "07b7fe3136a162b8d2ee47994ac4629b"
server_key_version: "encryptionkey@0"
I20260928 08:43:46.497228 635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d0c974123bb74df18593b275e93225d4"
format_stamp: "Formatted at 2026-09-28 08:43:46 on dist-test-slave-n6xs"
server_key: "c8b0af2ba13daa5e89519a01f3bec4aa"
server_key_iv: "07b7fe3136a162b8d2ee47994ac4629b"
server_key_version: "encryptionkey@0"
I20260928 08:43:46.502503 635 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 08:43:46.506054 650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:43:46.507437 635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:43:46.507575 635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-1/wal
uuid: "d0c974123bb74df18593b275e93225d4"
format_stamp: "Formatted at 2026-09-28 08:43:46 on dist-test-slave-n6xs"
server_key: "c8b0af2ba13daa5e89519a01f3bec4aa"
server_key_iv: "07b7fe3136a162b8d2ee47994ac4629b"
server_key_version: "encryptionkey@0"
I20260928 08:43:46.507747 635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:43:46.532382 635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:43:46.537532 635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:43:46.537853 635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:43:46.538882 635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:43:46.540257 635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:43:46.540331 635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:43:46.540421 635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:43:46.540464 635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:43:46.576994 635 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33039
I20260928 08:43:46.577085 762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33039 every 8 connection(s)
I20260928 08:43:46.578326 635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:43:46.578881 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 635
I20260928 08:43:46.579057 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:43:46.579664 388 external_mini_cluster.cc:1428] Setting key e29a85018b178074a37bb02bd994ee80
Sep 28 08:43:46 dist-test-slave-n6xs krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260928 08:43:46.601019 765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:46.581244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52613 (local address 127.0.97.62:34887)
0928 08:43:46.581743 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:46.581749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:46.582905 (+ 1156us) server_negotiation.cc:407] Connection header received
0928 08:43:46.585062 (+ 2157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:46.585068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:46.585168 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:46.585295 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:46.587393 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:46.588235 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:46.589888 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:46.590463 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:46.590614 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:46.594625 (+ 4011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:46.594630 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:46.594633 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:46.594667 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:46.597603 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:46.598381 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:46.598390 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:46.598395 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:46.598555 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:46.599093 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:46.599097 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:46.599099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:46.599358 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:46.599482 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:46.600552 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0928 08:43:46.600731 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":257,"threads_started":1}
I20260928 08:43:46.602695 766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:46.581886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34887 (local address 127.0.97.2:52613)
0928 08:43:46.582734 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:46.582775 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:46.584758 (+ 1983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:46.585551 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:46.585560 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:46.586548 (+ 988us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:43:46.587192 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:46.587211 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:46.588401 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:46.588412 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:46.589702 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:46.589715 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:46.590708 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:46.590717 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:46.590975 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:46.591921 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:43:46.591973 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:43:46.594357 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:43:46.597763 (+ 3406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:46.597773 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:46.597799 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:46.598248 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:46.598683 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:46.598687 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:46.598693 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:46.598967 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:46.599510 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:43:46.599522 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:43:46.600077 (+ 555us) client_negotiation.cc:769] Sending connection context
0928 08:43:46.600419 (+ 342us) client_negotiation.cc:240] Negotiation successful
0928 08:43:46.601122 (+ 703us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":387,"threads_started":1}
I20260928 08:43:46.604405 763 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34887
I20260928 08:43:46.604740 763 heartbeater.cc:464] Registering TS with master...
I20260928 08:43:46.605436 763 heartbeater.cc:511] Master 127.0.97.62:34887 requested a full tablet report, sending...
I20260928 08:43:46.606877 440 ts_manager.cc:194] Registered new tserver with Master: d0c974123bb74df18593b275e93225d4 (127.0.97.2:33039)
I20260928 08:43:46.607725 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52613
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:43:46.621538 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34887
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:43:46.980108 769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:43:46.980397 769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:43:46.980558 769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:43:46.988456 769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:43:46.988579 769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:43:46.988744 769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:43:46.999308 769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34887
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:43:47.000692 769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:43:47.002035 769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:43:47.016201 774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:47.016222 775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:47.016162 777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:43:47.017256 769 server_base.cc:1034] running on GCE node
I20260928 08:43:47.017926 769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:43:47.018903 769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:43:47.020097 769 hybrid_clock.cc:648] HybridClock initialized: now 1790585027020052 us; error 33 us; skew 500 ppm
I20260928 08:43:47.020416 769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:43:47 dist-test-slave-n6xs krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:43:47.023528 769 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:43:47.025177 769 webserver.cc:460] Webserver started at http://127.0.97.3:38457/ using document root <none> and password file <none>
I20260928 08:43:47.025893 769 fs_manager.cc:359] Metadata directory not provided
I20260928 08:43:47.026046 769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:43:47.026366 769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:43:47.029361 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-2/data/instance:
uuid: "a63092f07cd9432284f73990d4936c33"
format_stamp: "Formatted at 2026-09-28 08:43:47 on dist-test-slave-n6xs"
server_key: "3f690c119f6e1aa2ac982f29dde5daf8"
server_key_iv: "4769d7623d05a93d5fde5d2c592d8271"
server_key_version: "encryptionkey@0"
I20260928 08:43:47.030181 769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a63092f07cd9432284f73990d4936c33"
format_stamp: "Formatted at 2026-09-28 08:43:47 on dist-test-slave-n6xs"
server_key: "3f690c119f6e1aa2ac982f29dde5daf8"
server_key_iv: "4769d7623d05a93d5fde5d2c592d8271"
server_key_version: "encryptionkey@0"
I20260928 08:43:47.035439 769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260928 08:43:47.038879 784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:43:47.040333 769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:43:47.040508 769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-2/wal
uuid: "a63092f07cd9432284f73990d4936c33"
format_stamp: "Formatted at 2026-09-28 08:43:47 on dist-test-slave-n6xs"
server_key: "3f690c119f6e1aa2ac982f29dde5daf8"
server_key_iv: "4769d7623d05a93d5fde5d2c592d8271"
server_key_version: "encryptionkey@0"
I20260928 08:43:47.040725 769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:43:47.056581 769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:43:47.061277 769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:43:47.061550 769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:43:47.062453 769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:43:47.063745 769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:43:47.063887 769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 08:43:47.064003 769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:43:47.064055 769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:43:47.089649 629 heartbeater.cc:503] Master 127.0.97.62:34887 was elected leader, sending a full tablet report...
I20260928 08:43:47.099858 769 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:35403
I20260928 08:43:47.099911 896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:35403 every 8 connection(s)
I20260928 08:43:47.101258 769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:43:47.110005 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 769
I20260928 08:43:47.110193 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790585022442112-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:43:47.110754 388 external_mini_cluster.cc:1428] Setting key 1543263bb544308886b20503f7cff0d2
Sep 28 08:43:47 dist-test-slave-n6xs krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:43:47.122573 899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:47.103792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45451 (local address 127.0.97.62:34887)
0928 08:43:47.104675 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:47.104680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:47.105389 (+ 709us) server_negotiation.cc:407] Connection header received
0928 08:43:47.107203 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:47.107212 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:47.107273 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:47.107369 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:47.108962 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:47.109728 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:47.112189 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:47.112758 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:47.112856 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:47.117130 (+ 4274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:47.117135 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:47.117139 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:47.117169 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:47.119800 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:47.120543 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:47.120547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:47.120549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:47.120659 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:47.121068 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:47.121072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:47.121074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:47.121319 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:47.121425 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:47.122131 (+ 706us) server_negotiation.cc:299] Negotiation successful
0928 08:43:47.122291 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":603,"threads_started":1}
I20260928 08:43:47.124010 900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:47.104501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34887 (local address 127.0.97.3:45451)
0928 08:43:47.105223 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:47.105267 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:47.106951 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:47.107594 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:47.107603 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:47.108239 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:43:47.108800 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:47.108818 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:47.109892 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:47.109898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:47.112011 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:47.112027 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:47.112999 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:47.113008 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:47.113343 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:47.114059 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:43:47.114093 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:43:47.116868 (+ 2775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:43:47.120011 (+ 3143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:47.120017 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:47.120031 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:47.120431 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:47.120758 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:47.120761 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:47.120763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:47.120931 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:47.121427 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:43:47.121433 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:43:47.121752 (+ 319us) client_negotiation.cc:769] Sending connection context
0928 08:43:47.122047 (+ 295us) client_negotiation.cc:240] Negotiation successful
0928 08:43:47.122436 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":264,"threads_started":1}
I20260928 08:43:47.125468 897 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34887
I20260928 08:43:47.125813 897 heartbeater.cc:464] Registering TS with master...
I20260928 08:43:47.126734 897 heartbeater.cc:511] Master 127.0.97.62:34887 requested a full tablet report, sending...
I20260928 08:43:47.128170 440 ts_manager.cc:194] Registered new tserver with Master: a63092f07cd9432284f73990d4936c33 (127.0.97.3:35403)
I20260928 08:43:47.128962 440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45451
I20260928 08:43:47.139714 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 08:43:47 dist-test-slave-n6xs krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585027, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 08:43:47 dist-test-slave-n6xs krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585027, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:43:47.185094 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:47.169190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35594 (local address 127.0.97.1:35207)
0928 08:43:47.169979 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:47.169987 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:47.170018 (+ 31us) server_negotiation.cc:407] Connection header received
0928 08:43:47.170077 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:47.170083 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:47.170294 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:47.170445 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:47.171926 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:47.172937 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:47.174201 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:47.174784 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:47.174885 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:47.179382 (+ 4497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:47.179396 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:47.179412 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:47.179451 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:47.182182 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:47.182877 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:47.182885 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:47.182887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:47.183012 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:47.183397 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:47.183403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:47.183406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:47.183767 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:47.184129 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:47.184453 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 08:43:47.184740 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":514,"threads_started":1}
W20260928 08:43:47.186419 544 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35594
Sep 28 08:43:47 dist-test-slave-n6xs krb5kdc[395](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 08:43:47 dist-test-slave-n6xs krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:43:47.230988 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:47.217130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35610 (local address 127.0.97.1:35207)
0928 08:43:47.217369 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:47.217378 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:47.217706 (+ 328us) server_negotiation.cc:407] Connection header received
0928 08:43:47.217905 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:47.217911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:47.218069 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:47.218184 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:47.219478 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:47.220318 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:47.221495 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:47.221880 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:47.221989 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:47.225875 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:47.225880 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:47.225883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:47.225919 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:47.228851 (+ 2932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:47.229463 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:47.229467 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:47.229469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:47.229579 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:47.229869 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:47.229872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:47.229874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:47.230112 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:47.230270 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:47.230512 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 08:43:47.230673 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 08:43:47.232605 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 501
I20260928 08:43:47.245012 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:47.233622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35624 (local address 127.0.97.1:35207)
0928 08:43:47.233828 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:47.233834 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:47.233947 (+ 113us) server_negotiation.cc:407] Connection header received
0928 08:43:47.234070 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:47.234075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:47.234132 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:47.234210 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:47.235397 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:47.236149 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:47.237376 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:47.237772 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:47.237864 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:47.240266 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:47.240271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:47.240274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:47.240304 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:47.242624 (+ 2320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:47.243239 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:47.243242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:47.243245 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:47.243364 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:47.243657 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:47.243660 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:47.243662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:47.243981 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:47.244204 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:47.244465 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 08:43:47.244660 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 08:43:47.245894 624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:43:47.329049 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 501
I20260928 08:43:47.345660 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 635
Sep 28 08:43:47 dist-test-slave-n6xs krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 08:43:47.362594 912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:47.346596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35090 (local address 127.0.97.2:33039)
0928 08:43:47.347199 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:47.347206 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:47.347232 (+ 26us) server_negotiation.cc:407] Connection header received
0928 08:43:47.347292 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:47.347299 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:47.347511 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:47.347642 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:47.348895 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:47.349936 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:47.351184 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:47.351737 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:47.351917 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:47.356800 (+ 4883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:47.356825 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:47.356842 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:47.356883 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:47.359943 (+ 3060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:47.360626 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:47.360634 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:47.360636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:47.360760 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:47.361131 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:47.361133 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:47.361136 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:47.361486 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:47.361721 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:47.362051 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 08:43:47.362289 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":287,"threads_started":1}
I20260928 08:43:47.364041 758 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:43:47.466984 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 635
I20260928 08:43:47.489326 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 769
Sep 28 08:43:47 dist-test-slave-n6xs krb5kdc[395](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 08:43:47.505605 914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:47.490330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58782 (local address 127.0.97.3:35403)
0928 08:43:47.490956 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:47.490963 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:47.490994 (+ 31us) server_negotiation.cc:407] Connection header received
0928 08:43:47.491060 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:47.491066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:47.491282 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:47.491478 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:47.492790 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:47.493808 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:47.495092 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:47.495652 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:47.495785 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:47.499910 (+ 4125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:47.499922 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:47.499938 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:47.499971 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:47.502779 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:47.503437 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:47.503444 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:47.503446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:47.503564 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:47.503977 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:47.503984 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:47.503987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:47.504343 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:47.504646 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:47.504973 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 08:43:47.505253 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":316,"threads_started":1}
I20260928 08:43:47.506884 892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:43:47.589769 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 769
I20260928 08:43:47.605690 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 408
I20260928 08:43:47.606902 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:43:47.708442 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 408
2026-09-28T08:43:47Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5229 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:43:47 dist-test-slave-n6xs krb5kdc[919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:43:47 dist-test-slave-n6xs krb5kdc[919](info): set up 2 sockets
Sep 28 08:43:47 dist-test-slave-n6xs krb5kdc[919](info): commencing operation
krb5kdc: starting...
W20260928 08:43:48.784021 388 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:43:48 dist-test-slave-n6xs krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:43:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:43:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:43:48.968098 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37019
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37019
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:43:49.292161 932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:43:49.292519 932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:43:49.292691 932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:43:49.300031 932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:43:49.300184 932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:43:49.300290 932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:43:49.300395 932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:43:49.300482 932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:43:49.310701 932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:43:49.312161 932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:43:49.313489 932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:43:49.325752 937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:49.325649 938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:49.325788 940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:43:49.326685 932 server_base.cc:1034] running on GCE node
I20260928 08:43:49.327612 932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:43:49.329255 932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:43:49.330475 932 hybrid_clock.cc:648] HybridClock initialized: now 1790585029330427 us; error 47 us; skew 500 ppm
I20260928 08:43:49.330871 932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:43:49 dist-test-slave-n6xs krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:43:49.339152 932 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:43:49.340947 932 webserver.cc:460] Webserver started at http://127.0.97.62:33847/ using document root <none> and password file <none>
I20260928 08:43:49.341738 932 fs_manager.cc:359] Metadata directory not provided
I20260928 08:43:49.341883 932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:43:49.342243 932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:43:49.345467 932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "6d1c3ed7ce3e4a33b995c284cf0572e8"
format_stamp: "Formatted at 2026-09-28 08:43:49 on dist-test-slave-n6xs"
server_key: "571d391ed923b01af5cd2722620db1f8"
server_key_iv: "e5a23d615ea0a24b6e778a90d60a5d31"
server_key_version: "encryptionkey@0"
I20260928 08:43:49.346272 932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "6d1c3ed7ce3e4a33b995c284cf0572e8"
format_stamp: "Formatted at 2026-09-28 08:43:49 on dist-test-slave-n6xs"
server_key: "571d391ed923b01af5cd2722620db1f8"
server_key_iv: "e5a23d615ea0a24b6e778a90d60a5d31"
server_key_version: "encryptionkey@0"
I20260928 08:43:49.351778 932 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 08:43:49.355748 947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:43:49.357270 932 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 08:43:49.357462 932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "6d1c3ed7ce3e4a33b995c284cf0572e8"
format_stamp: "Formatted at 2026-09-28 08:43:49 on dist-test-slave-n6xs"
server_key: "571d391ed923b01af5cd2722620db1f8"
server_key_iv: "e5a23d615ea0a24b6e778a90d60a5d31"
server_key_version: "encryptionkey@0"
I20260928 08:43:49.357688 932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:43:49.380437 932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:43:49.385210 932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:43:49.385515 932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:43:49.404228 932 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37019
I20260928 08:43:49.404279 998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37019 every 8 connection(s)
I20260928 08:43:49.405709 932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:43:49.409416 999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:43:49.416057 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 932
I20260928 08:43:49.416265 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:43:49.417105 388 external_mini_cluster.cc:1428] Setting key 7d371334f3099a30dfe70d0848279bd2
I20260928 08:43:49.423157 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d1c3ed7ce3e4a33b995c284cf0572e8: Bootstrap starting.
I20260928 08:43:49.426227 999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d1c3ed7ce3e4a33b995c284cf0572e8: Neither blocks nor log segments found. Creating new log.
I20260928 08:43:49.427193 999 log.cc:824] T 00000000000000000000000000000000 P 6d1c3ed7ce3e4a33b995c284cf0572e8: Log is configured to *not* fsync() on all Append() calls
I20260928 08:43:49.429993 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d1c3ed7ce3e4a33b995c284cf0572e8: No bootstrap required, opened a new log
Sep 28 08:43:49 dist-test-slave-n6xs krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:43:49.438987 999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6d1c3ed7ce3e4a33b995c284cf0572e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1c3ed7ce3e4a33b995c284cf0572e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37019 } }
I20260928 08:43:49.439445 999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6d1c3ed7ce3e4a33b995c284cf0572e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:43:49.439580 999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6d1c3ed7ce3e4a33b995c284cf0572e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d1c3ed7ce3e4a33b995c284cf0572e8, State: Initialized, Role: FOLLOWER
I20260928 08:43:49.440214 999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d1c3ed7ce3e4a33b995c284cf0572e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1c3ed7ce3e4a33b995c284cf0572e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37019 } }
I20260928 08:43:49.440451 999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6d1c3ed7ce3e4a33b995c284cf0572e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:43:49.440613 999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6d1c3ed7ce3e4a33b995c284cf0572e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:43:49.440650 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:49.420005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45876 (local address 127.0.97.62:37019)
0928 08:43:49.420786 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:49.420798 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:49.420841 (+ 43us) server_negotiation.cc:407] Connection header received
0928 08:43:49.422521 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:49.422551 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:49.422947 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:49.423333 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:49.424846 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:49.426108 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:49.427455 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:49.428358 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:49.428561 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:49.432791 (+ 4230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:49.432799 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:49.432813 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:49.432851 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:49.436111 (+ 3260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:49.436897 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:49.436903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:49.436909 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:49.437118 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:49.437552 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:49.437555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:49.437557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:49.438110 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:49.438323 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:49.438845 (+ 522us) server_negotiation.cc:299] Negotiation successful
0928 08:43:49.439270 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":330,"threads_started":1}
I20260928 08:43:49.440814 999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6d1c3ed7ce3e4a33b995c284cf0572e8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:43:49.443142 999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6d1c3ed7ce3e4a33b995c284cf0572e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1c3ed7ce3e4a33b995c284cf0572e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37019 } }
I20260928 08:43:49.443668 999 leader_election.cc:302] T 00000000000000000000000000000000 P 6d1c3ed7ce3e4a33b995c284cf0572e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d1c3ed7ce3e4a33b995c284cf0572e8; no voters:
I20260928 08:43:49.444227 999 leader_election.cc:288] T 00000000000000000000000000000000 P 6d1c3ed7ce3e4a33b995c284cf0572e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:43:49.444365 1004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6d1c3ed7ce3e4a33b995c284cf0572e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:43:49.444998 1004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6d1c3ed7ce3e4a33b995c284cf0572e8 [term 1 LEADER]: Becoming Leader. State: Replica: 6d1c3ed7ce3e4a33b995c284cf0572e8, State: Running, Role: LEADER
I20260928 08:43:49.445536 1004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d1c3ed7ce3e4a33b995c284cf0572e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1c3ed7ce3e4a33b995c284cf0572e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37019 } }
I20260928 08:43:49.446254 999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d1c3ed7ce3e4a33b995c284cf0572e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:43:49.448637 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d1c3ed7ce3e4a33b995c284cf0572e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d1c3ed7ce3e4a33b995c284cf0572e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1c3ed7ce3e4a33b995c284cf0572e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37019 } } }
I20260928 08:43:49.448599 1006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d1c3ed7ce3e4a33b995c284cf0572e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d1c3ed7ce3e4a33b995c284cf0572e8. Latest consensus state: current_term: 1 leader_uuid: "6d1c3ed7ce3e4a33b995c284cf0572e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d1c3ed7ce3e4a33b995c284cf0572e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37019 } } }
I20260928 08:43:49.449045 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d1c3ed7ce3e4a33b995c284cf0572e8 [sys.catalog]: This master's current role is: LEADER
I20260928 08:43:49.449043 1006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d1c3ed7ce3e4a33b995c284cf0572e8 [sys.catalog]: This master's current role is: LEADER
I20260928 08:43:49.449925 1010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:43:49.454424 1010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:43:49.465137 1010 catalog_manager.cc:1347] Generated new cluster ID: 428cb7b71ca549dd962745ad7a02a53c
I20260928 08:43:49.465366 1010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:43:49.487957 1010 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:43:49.489302 1010 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:43:49.495744 1010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6d1c3ed7ce3e4a33b995c284cf0572e8: Generated new TSK 0
I20260928 08:43:49.496592 1010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:43:49.544940 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37019
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:43:49.886418 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:43:49.886722 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:43:49.886821 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:43:49.894217 1025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:43:49.894333 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:43:49.894457 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:43:49.904603 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37019
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:43:49.905915 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:43:49.907300 1025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:43:49.920310 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:49.920284 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:49.920878 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:43:49.921474 1025 server_base.cc:1034] running on GCE node
I20260928 08:43:49.922212 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:43:49.923223 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:43:49.924430 1025 hybrid_clock.cc:648] HybridClock initialized: now 1790585029924389 us; error 44 us; skew 500 ppm
I20260928 08:43:49.924784 1025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:43:49 dist-test-slave-n6xs krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:43:49.928169 1025 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:43:49.929713 1025 webserver.cc:460] Webserver started at http://127.0.97.1:46471/ using document root <none> and password file <none>
I20260928 08:43:49.930433 1025 fs_manager.cc:359] Metadata directory not provided
I20260928 08:43:49.930578 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:43:49.930884 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:43:49.933830 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-0/data/instance:
uuid: "318b3fd57bb4471292a62f7526372a2f"
format_stamp: "Formatted at 2026-09-28 08:43:49 on dist-test-slave-n6xs"
server_key: "6e401f447193fe63724ee5ab0cb13c98"
server_key_iv: "de95f518b312104d62a771dd377d1d32"
server_key_version: "encryptionkey@0"
I20260928 08:43:49.934646 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-0/wal/instance:
uuid: "318b3fd57bb4471292a62f7526372a2f"
format_stamp: "Formatted at 2026-09-28 08:43:49 on dist-test-slave-n6xs"
server_key: "6e401f447193fe63724ee5ab0cb13c98"
server_key_iv: "de95f518b312104d62a771dd377d1d32"
server_key_version: "encryptionkey@0"
I20260928 08:43:49.940285 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 08:43:49.944120 1040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:43:49.945580 1025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:43:49.945756 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-0/wal
uuid: "318b3fd57bb4471292a62f7526372a2f"
format_stamp: "Formatted at 2026-09-28 08:43:49 on dist-test-slave-n6xs"
server_key: "6e401f447193fe63724ee5ab0cb13c98"
server_key_iv: "de95f518b312104d62a771dd377d1d32"
server_key_version: "encryptionkey@0"
I20260928 08:43:49.945964 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:43:49.978955 1025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:43:49.983265 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:43:49.983584 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:43:49.984596 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:43:49.985940 1025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:43:49.986054 1025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:43:49.986213 1025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:43:49.986286 1025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:43:50.019577 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38927
I20260928 08:43:50.019657 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38927 every 8 connection(s)
I20260928 08:43:50.020982 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:43:50.024173 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 1025
I20260928 08:43:50.024367 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:43:50.024981 388 external_mini_cluster.cc:1428] Setting key 446a356e5bb9d4495864cf81269b16b2
Sep 28 08:43:50 dist-test-slave-n6xs krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 08:43:50.043610 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:50.023513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37553 (local address 127.0.97.62:37019)
0928 08:43:50.024031 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:50.024038 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:50.025139 (+ 1101us) server_negotiation.cc:407] Connection header received
0928 08:43:50.027537 (+ 2398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:50.027542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:50.027630 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:50.027752 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:50.029420 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:50.030579 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:50.032476 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:50.032871 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:50.032971 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:50.037566 (+ 4595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:50.037571 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:50.037574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:50.037605 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:50.040292 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:50.041244 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:50.041248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:50.041250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:50.041359 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:50.041884 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:50.041887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:50.041888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:50.042136 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:50.042240 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:50.043118 (+ 878us) server_negotiation.cc:299] Negotiation successful
0928 08:43:50.043306 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":247,"threads_started":1}
I20260928 08:43:50.044775 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:50.024132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37019 (local address 127.0.97.1:37553)
0928 08:43:50.024966 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:50.025017 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:50.027267 (+ 2250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:50.028110 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:50.028118 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:50.028649 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:43:50.029237 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:50.029255 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:50.030772 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:50.030777 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:50.032315 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:50.032333 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:50.033106 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:50.033116 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:50.033466 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:50.034227 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:43:50.034265 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:43:50.037307 (+ 3042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:43:50.040440 (+ 3133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:50.040452 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:50.040475 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:50.041115 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:50.041470 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:50.041476 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:50.041481 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:50.041765 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:50.042278 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:43:50.042289 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:43:50.042738 (+ 449us) client_negotiation.cc:769] Sending connection context
0928 08:43:50.042986 (+ 248us) client_negotiation.cc:240] Negotiation successful
0928 08:43:50.043460 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":380,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260928 08:43:50.046612 1153 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37019
I20260928 08:43:50.046941 1153 heartbeater.cc:464] Registering TS with master...
I20260928 08:43:50.047767 1153 heartbeater.cc:511] Master 127.0.97.62:37019 requested a full tablet report, sending...
I20260928 08:43:50.050071 964 ts_manager.cc:194] Registered new tserver with Master: 318b3fd57bb4471292a62f7526372a2f (127.0.97.1:38927)
I20260928 08:43:50.052323 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:37553
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 08:43:50.068322 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37019
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:43:50.391062 1159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:43:50.391368 1159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:43:50.391471 1159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:43:50.398626 1159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:43:50.398751 1159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:43:50.398860 1159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:43:50.408774 1159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37019
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:43:50.410097 1159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:43:50.411621 1159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:43:50.425100 1165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:50.425153 1164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:50.425083 1167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:43:50.425930 1159 server_base.cc:1034] running on GCE node
I20260928 08:43:50.426503 1159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:43:50.427428 1159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:43:50.428630 1159 hybrid_clock.cc:648] HybridClock initialized: now 1790585030428558 us; error 61 us; skew 500 ppm
I20260928 08:43:50.428912 1159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:43:50 dist-test-slave-n6xs krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:43:50.432089 1159 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:43:50.433486 1159 webserver.cc:460] Webserver started at http://127.0.97.2:35069/ using document root <none> and password file <none>
I20260928 08:43:50.434168 1159 fs_manager.cc:359] Metadata directory not provided
I20260928 08:43:50.434274 1159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:43:50.434607 1159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:43:50.437577 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-1/data/instance:
uuid: "66cf858e3f40435ba2d5fb1bd9f8f8fe"
format_stamp: "Formatted at 2026-09-28 08:43:50 on dist-test-slave-n6xs"
server_key: "78b755b444d5b823308040c794a346e1"
server_key_iv: "400efd48fee58da20324f0d68d3723cc"
server_key_version: "encryptionkey@0"
I20260928 08:43:50.438324 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-1/wal/instance:
uuid: "66cf858e3f40435ba2d5fb1bd9f8f8fe"
format_stamp: "Formatted at 2026-09-28 08:43:50 on dist-test-slave-n6xs"
server_key: "78b755b444d5b823308040c794a346e1"
server_key_iv: "400efd48fee58da20324f0d68d3723cc"
server_key_version: "encryptionkey@0"
I20260928 08:43:50.443607 1159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 08:43:50.447273 1174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:43:50.448904 1159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:43:50.449082 1159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-1/wal
uuid: "66cf858e3f40435ba2d5fb1bd9f8f8fe"
format_stamp: "Formatted at 2026-09-28 08:43:50 on dist-test-slave-n6xs"
server_key: "78b755b444d5b823308040c794a346e1"
server_key_iv: "400efd48fee58da20324f0d68d3723cc"
server_key_version: "encryptionkey@0"
I20260928 08:43:50.449266 1159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:43:50.493084 1159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:43:50.497522 1159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:43:50.497933 1159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:43:50.498920 1159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:43:50.500352 1159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:43:50.500423 1159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:43:50.500567 1159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:43:50.500619 1159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:43:50.533638 1159 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36491
I20260928 08:43:50.533681 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36491 every 8 connection(s)
I20260928 08:43:50.534924 1159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:43:50.536549 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 1159
I20260928 08:43:50.536752 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:43:50.537227 388 external_mini_cluster.cc:1428] Setting key 529d7f9e6eff92091aaa6aedbe896ccb
Sep 28 08:43:50 dist-test-slave-n6xs krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260928 08:43:50.560012 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:50.537901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44579 (local address 127.0.97.62:37019)
0928 08:43:50.538110 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:50.538116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:50.539476 (+ 1360us) server_negotiation.cc:407] Connection header received
0928 08:43:50.542554 (+ 3078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:50.542558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:50.542654 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:50.542758 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:50.544873 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:50.545701 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:50.547559 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:50.548407 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:50.548558 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:50.552792 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:50.552800 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:50.552805 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:50.552850 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:50.556234 (+ 3384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:50.557305 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:50.557309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:50.557311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:50.557452 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:50.557872 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:50.557875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:50.557877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:50.558201 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:50.558353 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:50.559216 (+ 863us) server_negotiation.cc:299] Negotiation successful
0928 08:43:50.559503 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 08:43:50.560667 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:50.538412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37019 (local address 127.0.97.2:44579)
0928 08:43:50.539300 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:50.539349 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:50.542189 (+ 2840us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:50.543115 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:50.543131 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:50.543967 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:43:50.544686 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:50.544703 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:50.545900 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:50.545913 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:50.547401 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:50.547412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:50.548638 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:50.548644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:50.548899 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:50.549716 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:43:50.549755 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:43:50.552476 (+ 2721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:43:50.556381 (+ 3905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:50.556388 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:50.556402 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:50.557169 (+ 767us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:50.557576 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:50.557581 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:50.557583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:50.557770 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:50.558428 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:43:50.558439 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:43:50.558817 (+ 378us) client_negotiation.cc:769] Sending connection context
0928 08:43:50.559048 (+ 231us) client_negotiation.cc:240] Negotiation successful
0928 08:43:50.559440 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":410,"threads_started":1}
I20260928 08:43:50.562332 1287 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37019
I20260928 08:43:50.562670 1287 heartbeater.cc:464] Registering TS with master...
I20260928 08:43:50.563414 1287 heartbeater.cc:511] Master 127.0.97.62:37019 requested a full tablet report, sending...
I20260928 08:43:50.564910 964 ts_manager.cc:194] Registered new tserver with Master: 66cf858e3f40435ba2d5fb1bd9f8f8fe (127.0.97.2:36491)
I20260928 08:43:50.565686 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:44579
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:43:50.580745 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37019
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:43:50.905422 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:43:50.905827 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:43:50.905972 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:43:50.914166 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:43:50.914294 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:43:50.914409 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:43:50.924336 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37019
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:43:50.925633 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:43:50.927073 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:43:50.940443 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:50.940423 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:50.940482 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:43:50.941254 1292 server_base.cc:1034] running on GCE node
I20260928 08:43:50.941886 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:43:50.942925 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:43:50.944175 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790585030944136 us; error 33 us; skew 500 ppm
I20260928 08:43:50.944530 1292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:43:50 dist-test-slave-n6xs krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:43:50.948210 1292 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:43:50.949854 1292 webserver.cc:460] Webserver started at http://127.0.97.3:41281/ using document root <none> and password file <none>
I20260928 08:43:50.950641 1292 fs_manager.cc:359] Metadata directory not provided
I20260928 08:43:50.950788 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:43:50.951113 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:43:50.954391 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-2/data/instance:
uuid: "ca97767caa6c4d33b342a39661fe6bf0"
format_stamp: "Formatted at 2026-09-28 08:43:50 on dist-test-slave-n6xs"
server_key: "f083e29f36e0330970db278c5fc313b7"
server_key_iv: "d17e767b1a3f0aa703a04c3fa8943c37"
server_key_version: "encryptionkey@0"
I20260928 08:43:50.955195 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ca97767caa6c4d33b342a39661fe6bf0"
format_stamp: "Formatted at 2026-09-28 08:43:50 on dist-test-slave-n6xs"
server_key: "f083e29f36e0330970db278c5fc313b7"
server_key_iv: "d17e767b1a3f0aa703a04c3fa8943c37"
server_key_version: "encryptionkey@0"
I20260928 08:43:50.961259 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.001s
I20260928 08:43:50.965201 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:43:50.966791 1292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:43:50.966995 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-2/wal
uuid: "ca97767caa6c4d33b342a39661fe6bf0"
format_stamp: "Formatted at 2026-09-28 08:43:50 on dist-test-slave-n6xs"
server_key: "f083e29f36e0330970db278c5fc313b7"
server_key_iv: "d17e767b1a3f0aa703a04c3fa8943c37"
server_key_version: "encryptionkey@0"
I20260928 08:43:50.967203 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:43:51.001814 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:43:51.006595 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:43:51.006903 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:43:51.008001 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:43:51.009488 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:43:51.009562 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:43:51.009712 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:43:51.009766 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:43:51.044786 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39181
I20260928 08:43:51.044826 1419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39181 every 8 connection(s)
I20260928 08:43:51.046197 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:43:51.048774 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 1292
I20260928 08:43:51.048961 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790585022442112-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:43:51.049451 388 external_mini_cluster.cc:1428] Setting key daa9c8b51cca19235af10da675e9399d
I20260928 08:43:51.055470 1153 heartbeater.cc:503] Master 127.0.97.62:37019 was elected leader, sending a full tablet report...
Sep 28 08:43:51 dist-test-slave-n6xs krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:43:51.068244 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:51.049000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36807 (local address 127.0.97.62:37019)
0928 08:43:51.049233 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:51.049239 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:51.050671 (+ 1432us) server_negotiation.cc:407] Connection header received
0928 08:43:51.053163 (+ 2492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:51.053167 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:51.053237 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:51.053310 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:51.054819 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.055687 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.057706 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.058100 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.058203 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:51.061948 (+ 3745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:51.061953 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:51.061957 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:51.061990 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:51.065397 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:51.066211 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:51.066214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:51.066217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:51.066333 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:51.066714 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:51.066717 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:51.066719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:51.066962 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:51.067068 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:51.067742 (+ 674us) server_negotiation.cc:299] Negotiation successful
0928 08:43:51.067981 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 08:43:51.069247 1422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:51.049596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37019 (local address 127.0.97.3:36807)
0928 08:43:51.050505 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:51.050561 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:51.052880 (+ 2319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:51.053526 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:51.053535 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:51.054083 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:43:51.054660 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:51.054677 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.055876 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.055880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:51.057567 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:51.057578 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.058287 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.058291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:51.058480 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:51.059186 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:43:51.059211 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:43:51.061726 (+ 2515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:43:51.065591 (+ 3865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:51.065599 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:51.065614 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:51.066099 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:51.066426 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:51.066429 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:51.066431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:51.066615 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:51.067061 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:43:51.067078 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:43:51.067408 (+ 330us) client_negotiation.cc:769] Sending connection context
0928 08:43:51.067643 (+ 235us) client_negotiation.cc:240] Negotiation successful
0928 08:43:51.068148 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":405,"threads_started":1}
I20260928 08:43:51.070636 1420 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37019
I20260928 08:43:51.070946 1420 heartbeater.cc:464] Registering TS with master...
I20260928 08:43:51.071745 1420 heartbeater.cc:511] Master 127.0.97.62:37019 requested a full tablet report, sending...
I20260928 08:43:51.073146 964 ts_manager.cc:194] Registered new tserver with Master: ca97767caa6c4d33b342a39661fe6bf0 (127.0.97.3:39181)
I20260928 08:43:51.073899 964 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36807
I20260928 08:43:51.077354 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:43:51.096241 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:51.083543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45890 (local address 127.0.97.62:37019)
0928 08:43:51.083712 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:51.083717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:51.084270 (+ 553us) server_negotiation.cc:407] Connection header received
0928 08:43:51.084403 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:51.084407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:51.084470 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:51.084560 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:51.086412 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.087123 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.088428 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.088819 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.088917 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:51.091190 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:51.091195 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:51.091198 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:51.091232 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:51.093899 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:51.094555 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:51.094558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:51.094560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:51.094683 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:51.094995 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:51.094998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:51.095000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:51.095285 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:51.095460 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:51.095772 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 08:43:51.096012 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 08:43:51.103070 964 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45890:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:43:51.110623 964 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:43:51.138056 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:51.124244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48984 (local address 127.0.97.2:36491)
0928 08:43:51.125562 (+ 1318us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:51.125572 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:51.127723 (+ 2151us) server_negotiation.cc:407] Connection header received
0928 08:43:51.127868 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:51.127874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:51.128113 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:51.128327 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:43:51.129488 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.130990 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.135310 (+ 4320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.136407 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.136571 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:51.136817 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:51.137489 (+ 672us) server_negotiation.cc:299] Negotiation successful
0928 08:43:51.137687 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1105,"thread_start_us":1544,"threads_started":1}
I20260928 08:43:51.138293 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:51.124015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36491 (local address 127.0.0.1:48984)
0928 08:43:51.127391 (+ 3376us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:51.127415 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:51.127562 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:51.128330 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:51.128335 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:51.128361 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:43:51.129274 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:51.129285 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.131195 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.131202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:51.135064 (+ 3862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:51.135080 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.136657 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.136664 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:51.136861 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:51.136943 (+ 82us) client_negotiation.cc:769] Sending connection context
0928 08:43:51.137097 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 08:43:51.137328 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3212,"mutex_wait_us":979,"thread_start_us":3037,"threads_started":1}
I20260928 08:43:51.138792 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:51.125601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48108 (local address 127.0.97.1:38927)
0928 08:43:51.127777 (+ 2176us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:51.127788 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:51.128232 (+ 444us) server_negotiation.cc:407] Connection header received
0928 08:43:51.129038 (+ 806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:51.129046 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:51.129287 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:51.129495 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:43:51.130269 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.131458 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.135245 (+ 3787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.136744 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.136862 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:51.137051 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:51.138285 (+ 1234us) server_negotiation.cc:299] Negotiation successful
0928 08:43:51.138474 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2003,"thread_start_us":1861,"threads_started":1}
I20260928 08:43:51.139542 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:51.124522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42996 (local address 127.0.97.3:39181)
0928 08:43:51.126157 (+ 1635us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:51.126168 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:51.126208 (+ 40us) server_negotiation.cc:407] Connection header received
0928 08:43:51.126336 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:51.126347 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:51.126691 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:51.128478 (+ 1787us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:43:51.129283 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.130991 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.135026 (+ 4035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.136575 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.136739 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:51.137025 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:51.139041 (+ 2016us) server_negotiation.cc:299] Negotiation successful
0928 08:43:51.139203 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1410,"thread_start_us":1228,"threads_started":1}
I20260928 08:43:51.140093 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:51.125197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38927 (local address 127.0.0.1:48108)
0928 08:43:51.128084 (+ 2887us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:51.128108 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:51.128225 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:51.129725 (+ 1500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:51.129729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:51.129754 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:43:51.130099 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:51.130107 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.131609 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.131614 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:51.135063 (+ 3449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:51.135080 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.137218 (+ 2138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.137223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:51.137458 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:51.137496 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 08:43:51.137590 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 08:43:51.137718 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2758,"mutex_wait_us":925,"thread_start_us":2623,"threads_started":1}
I20260928 08:43:51.140175 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:51.123690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39181 (local address 127.0.0.1:42996)
0928 08:43:51.124729 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:51.124765 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:51.124943 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:51.127998 (+ 3055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:51.128006 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:51.128038 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:43:51.129024 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:51.129043 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.131149 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.131155 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:51.134830 (+ 3675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:51.134849 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.137208 (+ 2359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.137215 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:51.137852 (+ 637us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:51.137883 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 08:43:51.137966 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 08:43:51.138110 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"mutex_wait_us":715,"thread_start_us":652,"threads_started":1}
I20260928 08:43:51.142876 1222 tablet_service.cc:1467] Processing CreateTablet for tablet d374c20f94824dedb03b6b11100c9efa (DEFAULT_TABLE table=test-table [id=48e100902c1a4c95be9eed1ff485a125]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:43:51.143076 1355 tablet_service.cc:1467] Processing CreateTablet for tablet d374c20f94824dedb03b6b11100c9efa (DEFAULT_TABLE table=test-table [id=48e100902c1a4c95be9eed1ff485a125]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:43:51.143641 1088 tablet_service.cc:1467] Processing CreateTablet for tablet d374c20f94824dedb03b6b11100c9efa (DEFAULT_TABLE table=test-table [id=48e100902c1a4c95be9eed1ff485a125]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:43:51.144667 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d374c20f94824dedb03b6b11100c9efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:43:51.144862 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d374c20f94824dedb03b6b11100c9efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:43:51.144893 1088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d374c20f94824dedb03b6b11100c9efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:43:51.156957 1438 tablet_bootstrap.cc:492] T d374c20f94824dedb03b6b11100c9efa P ca97767caa6c4d33b342a39661fe6bf0: Bootstrap starting.
I20260928 08:43:51.160296 1439 tablet_bootstrap.cc:492] T d374c20f94824dedb03b6b11100c9efa P 66cf858e3f40435ba2d5fb1bd9f8f8fe: Bootstrap starting.
I20260928 08:43:51.162073 1440 tablet_bootstrap.cc:492] T d374c20f94824dedb03b6b11100c9efa P 318b3fd57bb4471292a62f7526372a2f: Bootstrap starting.
I20260928 08:43:51.163304 1438 tablet_bootstrap.cc:654] T d374c20f94824dedb03b6b11100c9efa P ca97767caa6c4d33b342a39661fe6bf0: Neither blocks nor log segments found. Creating new log.
I20260928 08:43:51.164697 1438 log.cc:824] T d374c20f94824dedb03b6b11100c9efa P ca97767caa6c4d33b342a39661fe6bf0: Log is configured to *not* fsync() on all Append() calls
I20260928 08:43:51.165710 1440 tablet_bootstrap.cc:654] T d374c20f94824dedb03b6b11100c9efa P 318b3fd57bb4471292a62f7526372a2f: Neither blocks nor log segments found. Creating new log.
I20260928 08:43:51.165953 1439 tablet_bootstrap.cc:654] T d374c20f94824dedb03b6b11100c9efa P 66cf858e3f40435ba2d5fb1bd9f8f8fe: Neither blocks nor log segments found. Creating new log.
I20260928 08:43:51.167332 1439 log.cc:824] T d374c20f94824dedb03b6b11100c9efa P 66cf858e3f40435ba2d5fb1bd9f8f8fe: Log is configured to *not* fsync() on all Append() calls
I20260928 08:43:51.167639 1440 log.cc:824] T d374c20f94824dedb03b6b11100c9efa P 318b3fd57bb4471292a62f7526372a2f: Log is configured to *not* fsync() on all Append() calls
I20260928 08:43:51.168766 1438 tablet_bootstrap.cc:492] T d374c20f94824dedb03b6b11100c9efa P ca97767caa6c4d33b342a39661fe6bf0: No bootstrap required, opened a new log
I20260928 08:43:51.169076 1438 ts_tablet_manager.cc:1397] T d374c20f94824dedb03b6b11100c9efa P ca97767caa6c4d33b342a39661fe6bf0: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20260928 08:43:51.170943 1439 tablet_bootstrap.cc:492] T d374c20f94824dedb03b6b11100c9efa P 66cf858e3f40435ba2d5fb1bd9f8f8fe: No bootstrap required, opened a new log
I20260928 08:43:51.170945 1440 tablet_bootstrap.cc:492] T d374c20f94824dedb03b6b11100c9efa P 318b3fd57bb4471292a62f7526372a2f: No bootstrap required, opened a new log
I20260928 08:43:51.171216 1440 ts_tablet_manager.cc:1397] T d374c20f94824dedb03b6b11100c9efa P 318b3fd57bb4471292a62f7526372a2f: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260928 08:43:51.171216 1439 ts_tablet_manager.cc:1397] T d374c20f94824dedb03b6b11100c9efa P 66cf858e3f40435ba2d5fb1bd9f8f8fe: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260928 08:43:51.177057 1438 raft_consensus.cc:348] T d374c20f94824dedb03b6b11100c9efa P ca97767caa6c4d33b342a39661fe6bf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca97767caa6c4d33b342a39661fe6bf0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39181 } } peers { permanent_uuid: "66cf858e3f40435ba2d5fb1bd9f8f8fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36491 } } peers { permanent_uuid: "318b3fd57bb4471292a62f7526372a2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38927 } }
I20260928 08:43:51.177467 1438 raft_consensus.cc:374] T d374c20f94824dedb03b6b11100c9efa P ca97767caa6c4d33b342a39661fe6bf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:43:51.177565 1438 raft_consensus.cc:729] T d374c20f94824dedb03b6b11100c9efa P ca97767caa6c4d33b342a39661fe6bf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca97767caa6c4d33b342a39661fe6bf0, State: Initialized, Role: FOLLOWER
I20260928 08:43:51.178215 1438 consensus_queue.cc:260] T d374c20f94824dedb03b6b11100c9efa P ca97767caa6c4d33b342a39661fe6bf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca97767caa6c4d33b342a39661fe6bf0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39181 } } peers { permanent_uuid: "66cf858e3f40435ba2d5fb1bd9f8f8fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36491 } } peers { permanent_uuid: "318b3fd57bb4471292a62f7526372a2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38927 } }
I20260928 08:43:51.180505 1420 heartbeater.cc:503] Master 127.0.97.62:37019 was elected leader, sending a full tablet report...
I20260928 08:43:51.181227 1438 ts_tablet_manager.cc:1428] T d374c20f94824dedb03b6b11100c9efa P ca97767caa6c4d33b342a39661fe6bf0: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260928 08:43:51.182680 1439 raft_consensus.cc:348] T d374c20f94824dedb03b6b11100c9efa P 66cf858e3f40435ba2d5fb1bd9f8f8fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca97767caa6c4d33b342a39661fe6bf0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39181 } } peers { permanent_uuid: "66cf858e3f40435ba2d5fb1bd9f8f8fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36491 } } peers { permanent_uuid: "318b3fd57bb4471292a62f7526372a2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38927 } }
I20260928 08:43:51.182680 1440 raft_consensus.cc:348] T d374c20f94824dedb03b6b11100c9efa P 318b3fd57bb4471292a62f7526372a2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca97767caa6c4d33b342a39661fe6bf0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39181 } } peers { permanent_uuid: "66cf858e3f40435ba2d5fb1bd9f8f8fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36491 } } peers { permanent_uuid: "318b3fd57bb4471292a62f7526372a2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38927 } }
I20260928 08:43:51.183146 1439 raft_consensus.cc:374] T d374c20f94824dedb03b6b11100c9efa P 66cf858e3f40435ba2d5fb1bd9f8f8fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:43:51.183147 1440 raft_consensus.cc:374] T d374c20f94824dedb03b6b11100c9efa P 318b3fd57bb4471292a62f7526372a2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:43:51.183259 1439 raft_consensus.cc:729] T d374c20f94824dedb03b6b11100c9efa P 66cf858e3f40435ba2d5fb1bd9f8f8fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66cf858e3f40435ba2d5fb1bd9f8f8fe, State: Initialized, Role: FOLLOWER
I20260928 08:43:51.183249 1440 raft_consensus.cc:729] T d374c20f94824dedb03b6b11100c9efa P 318b3fd57bb4471292a62f7526372a2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 318b3fd57bb4471292a62f7526372a2f, State: Initialized, Role: FOLLOWER
I20260928 08:43:51.183933 1440 consensus_queue.cc:260] T d374c20f94824dedb03b6b11100c9efa P 318b3fd57bb4471292a62f7526372a2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca97767caa6c4d33b342a39661fe6bf0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39181 } } peers { permanent_uuid: "66cf858e3f40435ba2d5fb1bd9f8f8fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36491 } } peers { permanent_uuid: "318b3fd57bb4471292a62f7526372a2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38927 } }
I20260928 08:43:51.183928 1439 consensus_queue.cc:260] T d374c20f94824dedb03b6b11100c9efa P 66cf858e3f40435ba2d5fb1bd9f8f8fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca97767caa6c4d33b342a39661fe6bf0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39181 } } peers { permanent_uuid: "66cf858e3f40435ba2d5fb1bd9f8f8fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36491 } } peers { permanent_uuid: "318b3fd57bb4471292a62f7526372a2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38927 } }
I20260928 08:43:51.185566 1287 heartbeater.cc:503] Master 127.0.97.62:37019 was elected leader, sending a full tablet report...
I20260928 08:43:51.185909 1439 ts_tablet_manager.cc:1428] T d374c20f94824dedb03b6b11100c9efa P 66cf858e3f40435ba2d5fb1bd9f8f8fe: Time spent starting tablet: real 0.015s user 0.008s sys 0.008s
I20260928 08:43:51.186041 1440 ts_tablet_manager.cc:1428] T d374c20f94824dedb03b6b11100c9efa P 318b3fd57bb4471292a62f7526372a2f: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260928 08:43:51.230383 1444 raft_consensus.cc:482] T d374c20f94824dedb03b6b11100c9efa P ca97767caa6c4d33b342a39661fe6bf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:43:51.230742 1444 raft_consensus.cc:504] T d374c20f94824dedb03b6b11100c9efa P ca97767caa6c4d33b342a39661fe6bf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca97767caa6c4d33b342a39661fe6bf0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39181 } } peers { permanent_uuid: "66cf858e3f40435ba2d5fb1bd9f8f8fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36491 } } peers { permanent_uuid: "318b3fd57bb4471292a62f7526372a2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38927 } }
I20260928 08:43:51.232523 1444 leader_election.cc:288] T d374c20f94824dedb03b6b11100c9efa P ca97767caa6c4d33b342a39661fe6bf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66cf858e3f40435ba2d5fb1bd9f8f8fe (127.0.97.2:36491), 318b3fd57bb4471292a62f7526372a2f (127.0.97.1:38927)
I20260928 08:43:51.240015 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:51.232937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36017 (local address 127.0.97.1:38927)
0928 08:43:51.233115 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:51.233121 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:51.233253 (+ 132us) server_negotiation.cc:407] Connection header received
0928 08:43:51.233419 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:51.233427 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:51.233529 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:51.233645 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:43:51.234097 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.235169 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.237952 (+ 2783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.239134 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.239296 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:51.239420 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:51.239601 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 08:43:51.239753 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 08:43:51.240027 1422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:51.232865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38927 (local address 127.0.97.3:36017)
0928 08:43:51.233150 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:51.233167 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:51.233279 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:51.233694 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:51.233699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:51.233731 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:43:51.233947 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:51.233961 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.235292 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.235296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:51.237725 (+ 2429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:51.237742 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.239309 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.239313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:51.239443 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:51.239468 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 08:43:51.239544 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 08:43:51.239649 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":1}
I20260928 08:43:51.240633 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:51.232896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41911 (local address 127.0.97.2:36491)
0928 08:43:51.233098 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:51.233104 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:51.234065 (+ 961us) server_negotiation.cc:407] Connection header received
0928 08:43:51.234227 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:51.234234 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:51.234306 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:51.234413 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:43:51.234859 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.235686 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.238285 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.239598 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.239734 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:51.239853 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:51.240236 (+ 383us) server_negotiation.cc:299] Negotiation successful
0928 08:43:51.240388 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 08:43:51.241204 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:51.232875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36491 (local address 127.0.97.3:41911)
0928 08:43:51.233944 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:51.233960 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:51.234095 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:51.234419 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:51.234422 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:51.234469 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:43:51.234723 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:51.234729 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.236187 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.236192 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:51.238122 (+ 1930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:51.238131 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.239843 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.239851 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:51.240019 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:51.240074 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 08:43:51.240836 (+ 762us) client_negotiation.cc:240] Negotiation successful
0928 08:43:51.240944 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":936,"mutex_wait_us":282,"thread_start_us":426,"threads_started":1}
I20260928 08:43:51.241138 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d374c20f94824dedb03b6b11100c9efa" candidate_uuid: "ca97767caa6c4d33b342a39661fe6bf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "318b3fd57bb4471292a62f7526372a2f" is_pre_election: true
I20260928 08:43:51.241729 1108 raft_consensus.cc:2454] T d374c20f94824dedb03b6b11100c9efa P 318b3fd57bb4471292a62f7526372a2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca97767caa6c4d33b342a39661fe6bf0 in term 0.
I20260928 08:43:51.242144 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d374c20f94824dedb03b6b11100c9efa" candidate_uuid: "ca97767caa6c4d33b342a39661fe6bf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66cf858e3f40435ba2d5fb1bd9f8f8fe" is_pre_election: true
I20260928 08:43:51.242451 1308 leader_election.cc:302] T d374c20f94824dedb03b6b11100c9efa P ca97767caa6c4d33b342a39661fe6bf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 318b3fd57bb4471292a62f7526372a2f, ca97767caa6c4d33b342a39661fe6bf0; no voters:
I20260928 08:43:51.242700 1242 raft_consensus.cc:2454] T d374c20f94824dedb03b6b11100c9efa P 66cf858e3f40435ba2d5fb1bd9f8f8fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca97767caa6c4d33b342a39661fe6bf0 in term 0.
I20260928 08:43:51.242911 1444 raft_consensus.cc:2781] T d374c20f94824dedb03b6b11100c9efa P ca97767caa6c4d33b342a39661fe6bf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:43:51.243078 1444 raft_consensus.cc:482] T d374c20f94824dedb03b6b11100c9efa P ca97767caa6c4d33b342a39661fe6bf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:43:51.243209 1444 raft_consensus.cc:3037] T d374c20f94824dedb03b6b11100c9efa P ca97767caa6c4d33b342a39661fe6bf0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:43:51.245534 1444 raft_consensus.cc:504] T d374c20f94824dedb03b6b11100c9efa P ca97767caa6c4d33b342a39661fe6bf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca97767caa6c4d33b342a39661fe6bf0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39181 } } peers { permanent_uuid: "66cf858e3f40435ba2d5fb1bd9f8f8fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36491 } } peers { permanent_uuid: "318b3fd57bb4471292a62f7526372a2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38927 } }
I20260928 08:43:51.246199 1444 leader_election.cc:288] T d374c20f94824dedb03b6b11100c9efa P ca97767caa6c4d33b342a39661fe6bf0 [CANDIDATE]: Term 1 election: Requested vote from peers 66cf858e3f40435ba2d5fb1bd9f8f8fe (127.0.97.2:36491), 318b3fd57bb4471292a62f7526372a2f (127.0.97.1:38927)
I20260928 08:43:51.246712 1242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d374c20f94824dedb03b6b11100c9efa" candidate_uuid: "ca97767caa6c4d33b342a39661fe6bf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66cf858e3f40435ba2d5fb1bd9f8f8fe"
I20260928 08:43:51.246739 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d374c20f94824dedb03b6b11100c9efa" candidate_uuid: "ca97767caa6c4d33b342a39661fe6bf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "318b3fd57bb4471292a62f7526372a2f"
I20260928 08:43:51.246922 1242 raft_consensus.cc:3037] T d374c20f94824dedb03b6b11100c9efa P 66cf858e3f40435ba2d5fb1bd9f8f8fe [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:43:51.246963 1108 raft_consensus.cc:3037] T d374c20f94824dedb03b6b11100c9efa P 318b3fd57bb4471292a62f7526372a2f [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:43:51.249406 1108 raft_consensus.cc:2454] T d374c20f94824dedb03b6b11100c9efa P 318b3fd57bb4471292a62f7526372a2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca97767caa6c4d33b342a39661fe6bf0 in term 1.
I20260928 08:43:51.249881 1242 raft_consensus.cc:2454] T d374c20f94824dedb03b6b11100c9efa P 66cf858e3f40435ba2d5fb1bd9f8f8fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca97767caa6c4d33b342a39661fe6bf0 in term 1.
I20260928 08:43:51.250042 1308 leader_election.cc:302] T d374c20f94824dedb03b6b11100c9efa P ca97767caa6c4d33b342a39661fe6bf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 318b3fd57bb4471292a62f7526372a2f, ca97767caa6c4d33b342a39661fe6bf0; no voters:
I20260928 08:43:51.250372 1444 raft_consensus.cc:2781] T d374c20f94824dedb03b6b11100c9efa P ca97767caa6c4d33b342a39661fe6bf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:43:51.250815 1444 raft_consensus.cc:686] T d374c20f94824dedb03b6b11100c9efa P ca97767caa6c4d33b342a39661fe6bf0 [term 1 LEADER]: Becoming Leader. State: Replica: ca97767caa6c4d33b342a39661fe6bf0, State: Running, Role: LEADER
I20260928 08:43:51.251443 1444 consensus_queue.cc:237] T d374c20f94824dedb03b6b11100c9efa P ca97767caa6c4d33b342a39661fe6bf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca97767caa6c4d33b342a39661fe6bf0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39181 } } peers { permanent_uuid: "66cf858e3f40435ba2d5fb1bd9f8f8fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36491 } } peers { permanent_uuid: "318b3fd57bb4471292a62f7526372a2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38927 } }
I20260928 08:43:51.256733 963 catalog_manager.cc:5495] T d374c20f94824dedb03b6b11100c9efa P ca97767caa6c4d33b342a39661fe6bf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca97767caa6c4d33b342a39661fe6bf0 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "ca97767caa6c4d33b342a39661fe6bf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca97767caa6c4d33b342a39661fe6bf0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39181 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66cf858e3f40435ba2d5fb1bd9f8f8fe" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "318b3fd57bb4471292a62f7526372a2f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38927 } health_report { overall_health: UNKNOWN } } }
W20260928 08:43:51.273397 1154 tablet.cc:2369] T d374c20f94824dedb03b6b11100c9efa P 318b3fd57bb4471292a62f7526372a2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:43:51.287037 1288 tablet.cc:2369] T d374c20f94824dedb03b6b11100c9efa P 66cf858e3f40435ba2d5fb1bd9f8f8fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 08:43:51 dist-test-slave-n6xs krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
W20260928 08:43:51.298216 1421 tablet.cc:2369] T d374c20f94824dedb03b6b11100c9efa P ca97767caa6c4d33b342a39661fe6bf0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:43:51.301002 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:51.285161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48122 (local address 127.0.97.1:38927)
0928 08:43:51.285429 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:51.285438 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:51.285855 (+ 417us) server_negotiation.cc:407] Connection header received
0928 08:43:51.286080 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:51.286089 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:51.286154 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:51.286274 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:51.287848 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.288580 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.290193 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.290610 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.290744 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:51.294811 (+ 4067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:51.294826 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:51.294843 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:51.294878 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:51.297863 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:51.298786 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:51.298795 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:51.298799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:51.298976 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:51.299392 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:51.299396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:51.299399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:51.299769 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:51.300131 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:51.300410 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 08:43:51.300593 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
Sep 28 08:43:51 dist-test-slave-n6xs krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585031, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 08:43:51 dist-test-slave-n6xs krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585031, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:43:51.348873 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:51.334002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48134 (local address 127.0.97.1:38927)
0928 08:43:51.334251 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:51.334258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:51.334600 (+ 342us) server_negotiation.cc:407] Connection header received
0928 08:43:51.334762 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:51.334766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:51.334832 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:51.334924 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:51.336331 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.337095 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.338419 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.338857 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.339009 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:51.343372 (+ 4363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:51.343377 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:51.343381 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:51.343421 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:51.346187 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:51.347053 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:51.347058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:51.347060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:51.347201 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:51.347570 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:51.347574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:51.347576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:51.347914 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:51.348117 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:51.348352 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 08:43:51.348521 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
Sep 28 08:43:51 dist-test-slave-n6xs krb5kdc[919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 08:43:51 dist-test-slave-n6xs krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:43:51.396119 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:51.381468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48150 (local address 127.0.97.1:38927)
0928 08:43:51.381764 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:51.381771 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:51.381995 (+ 224us) server_negotiation.cc:407] Connection header received
0928 08:43:51.382251 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:51.382255 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:51.382317 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:51.382424 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:51.383796 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.384553 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.385819 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.386279 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.386457 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:51.390899 (+ 4442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:51.390905 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:51.390908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:51.390949 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:51.393413 (+ 2464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:51.394155 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:51.394161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:51.394163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:51.394281 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:51.394640 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:51.394647 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:51.394650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:51.394950 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:51.395215 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:51.395492 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 08:43:51.395744 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
W20260928 08:43:51.397915 1068 tablet_service.cc:2908] Rejecting scan request for tablet d374c20f94824dedb03b6b11100c9efa: Uninitialized: clean time has not yet been initialized
I20260928 08:43:51.401947 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 1025
I20260928 08:43:51.419222 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:51.402758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48156 (local address 127.0.97.1:38927)
0928 08:43:51.402924 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:51.402929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:51.403330 (+ 401us) server_negotiation.cc:407] Connection header received
0928 08:43:51.403464 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:51.403467 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:51.403525 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:51.403625 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:51.405221 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.405923 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.407591 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.408110 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.408223 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:51.411191 (+ 2968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:51.411197 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:51.411200 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:51.411238 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:51.415325 (+ 4087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:51.416306 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:51.416312 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:51.416316 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:51.416480 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:51.416933 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:51.416939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:51.416942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:51.417267 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:51.417551 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:51.418638 (+ 1087us) server_negotiation.cc:299] Negotiation successful
0928 08:43:51.418871 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 08:43:51.420120 1148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:43:51.512696 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 1025
I20260928 08:43:51.529791 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 1159
Sep 28 08:43:51 dist-test-slave-n6xs krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 08:43:51.545974 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:51.530603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48986 (local address 127.0.97.2:36491)
0928 08:43:51.530803 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:51.530810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:51.530830 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:43:51.531056 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:51.531063 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:51.531156 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:51.531262 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:51.532477 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.533312 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.534580 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.535098 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.535227 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:51.540064 (+ 4837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:51.540076 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:51.540091 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:51.540123 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:51.543087 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:51.544032 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:51.544038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:51.544041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:51.544157 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:51.544563 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:51.544566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:51.544568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:51.544949 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:51.545174 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:51.545502 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 08:43:51.545657 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 08:43:51.546603 1282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:43:51.635295 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 1159
I20260928 08:43:51.652127 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 1292
I20260928 08:43:51.657544 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:51.657062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38927 (local address 127.0.97.3:40121)
0928 08:43:51.657257 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:51.657404 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 08:43:51.657902 1308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38927: connect: Connection refused (error 111)
I20260928 08:43:51.659559 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:51.659177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38927 (local address 127.0.97.3:37325)
0928 08:43:51.659347 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:51.659428 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38927: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 08:43:51.659896 1308 consensus_peers.cc:487] T d374c20f94824dedb03b6b11100c9efa P ca97767caa6c4d33b342a39661fe6bf0 -> Peer 318b3fd57bb4471292a62f7526372a2f (127.0.97.1:38927): Couldn't send request to peer 318b3fd57bb4471292a62f7526372a2f. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38927: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
Sep 28 08:43:51 dist-test-slave-n6xs krb5kdc[919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 08:43:51.668185 1463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:51.652977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43010 (local address 127.0.97.3:39181)
0928 08:43:51.653482 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:51.653486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:51.653503 (+ 17us) server_negotiation.cc:407] Connection header received
0928 08:43:51.653558 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:51.653561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:51.653626 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:51.653728 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:51.654974 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.655785 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.657302 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:51.657808 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:51.657913 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:51.662674 (+ 4761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:51.662684 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:51.662699 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:51.662735 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:51.665501 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:51.666142 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:51.666149 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:51.666151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:51.666290 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:51.666614 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:51.666618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:51.666621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:51.667042 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:51.667241 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:51.667579 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 08:43:51.667812 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":288,"threads_started":1}
I20260928 08:43:51.668828 1415 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:43:51.759135 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 1292
I20260928 08:43:51.776087 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 932
I20260928 08:43:51.777333 994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:43:51.887246 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 932
2026-09-28T08:43:51Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4179 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:43:51 dist-test-slave-n6xs krb5kdc[1469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:43:51 dist-test-slave-n6xs krb5kdc[1469](info): set up 2 sockets
Sep 28 08:43:51 dist-test-slave-n6xs krb5kdc[1469](info): commencing operation
krb5kdc: starting...
W20260928 08:43:52.967790 388 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:43:53 dist-test-slave-n6xs krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:43:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:43:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:43:53.161803 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35969
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35969
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:43:53.486739 1482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:43:53.487174 1482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:43:53.487349 1482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:43:53.494675 1482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:43:53.494853 1482 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 08:43:53.494930 1482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:43:53.495064 1482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:43:53.495158 1482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:43:53.495245 1482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:43:53.505776 1482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35969
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:43:53.507244 1482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:43:53.508738 1482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:43:53.521607 1487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:53.521643 1490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:53.521590 1488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:43:53.522872 1482 server_base.cc:1034] running on GCE node
I20260928 08:43:53.523725 1482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:43:53.525286 1482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:43:53.526536 1482 hybrid_clock.cc:648] HybridClock initialized: now 1790585033526460 us; error 50 us; skew 500 ppm
I20260928 08:43:53.526934 1482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:43:53 dist-test-slave-n6xs krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:43:53.530403 1482 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:43:53.532110 1482 webserver.cc:460] Webserver started at http://127.0.97.62:33523/ using document root <none> and password file <none>
I20260928 08:43:53.532876 1482 fs_manager.cc:359] Metadata directory not provided
I20260928 08:43:53.532998 1482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:43:53.533355 1482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:43:53.536530 1482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "4d5d3dbb83324641a6fc9c439b1bd217"
format_stamp: "Formatted at 2026-09-28 08:43:53 on dist-test-slave-n6xs"
server_key: "d9d89835143e0c0e4793957a774a3bc6"
server_key_iv: "2326cc403bc665f6041665289d8b251b"
server_key_version: "encryptionkey@0"
I20260928 08:43:53.537410 1482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "4d5d3dbb83324641a6fc9c439b1bd217"
format_stamp: "Formatted at 2026-09-28 08:43:53 on dist-test-slave-n6xs"
server_key: "d9d89835143e0c0e4793957a774a3bc6"
server_key_iv: "2326cc403bc665f6041665289d8b251b"
server_key_version: "encryptionkey@0"
I20260928 08:43:53.542898 1482 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 08:43:53.546511 1497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:43:53.547919 1482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:43:53.548100 1482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "4d5d3dbb83324641a6fc9c439b1bd217"
format_stamp: "Formatted at 2026-09-28 08:43:53 on dist-test-slave-n6xs"
server_key: "d9d89835143e0c0e4793957a774a3bc6"
server_key_iv: "2326cc403bc665f6041665289d8b251b"
server_key_version: "encryptionkey@0"
I20260928 08:43:53.548322 1482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:43:53.581902 1482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:43:53.586596 1482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:43:53.586902 1482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:43:53.609645 1482 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35969
I20260928 08:43:53.609665 1558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35969 every 8 connection(s)
I20260928 08:43:53.611089 1482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:43:53.615485 1559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:43:53.620265 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 1482
I20260928 08:43:53.620482 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:43:53.621138 388 external_mini_cluster.cc:1428] Setting key f3f2b21f3e1426246db9bf505d6011ec
I20260928 08:43:53.627331 1559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d5d3dbb83324641a6fc9c439b1bd217: Bootstrap starting.
I20260928 08:43:53.631481 1559 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d5d3dbb83324641a6fc9c439b1bd217: Neither blocks nor log segments found. Creating new log.
I20260928 08:43:53.632814 1559 log.cc:824] T 00000000000000000000000000000000 P 4d5d3dbb83324641a6fc9c439b1bd217: Log is configured to *not* fsync() on all Append() calls
I20260928 08:43:53.636538 1559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d5d3dbb83324641a6fc9c439b1bd217: No bootstrap required, opened a new log
Sep 28 08:43:53 dist-test-slave-n6xs krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:43:53.645367 1559 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d5d3dbb83324641a6fc9c439b1bd217 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5d3dbb83324641a6fc9c439b1bd217" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35969 } }
I20260928 08:43:53.645849 1559 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d5d3dbb83324641a6fc9c439b1bd217 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:43:53.645998 1559 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d5d3dbb83324641a6fc9c439b1bd217 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d5d3dbb83324641a6fc9c439b1bd217, State: Initialized, Role: FOLLOWER
I20260928 08:43:53.646296 1562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:53.623227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49668 (local address 127.0.97.62:35969)
0928 08:43:53.624146 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:53.624159 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:53.624204 (+ 45us) server_negotiation.cc:407] Connection header received
0928 08:43:53.625974 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:53.625998 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:53.626412 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:53.626742 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:53.628750 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:53.630075 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:53.631955 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:53.632650 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:53.632788 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:53.637993 (+ 5205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:53.638001 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:53.638018 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:53.638058 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:53.641424 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:53.642249 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:53.642256 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:53.642264 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:53.642451 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:53.642869 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:53.642874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:53.642876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:53.643499 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:53.643855 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:53.644371 (+ 516us) server_negotiation.cc:299] Negotiation successful
0928 08:43:53.644890 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":393,"threads_started":1}
I20260928 08:43:53.646732 1559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d5d3dbb83324641a6fc9c439b1bd217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5d3dbb83324641a6fc9c439b1bd217" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35969 } }
I20260928 08:43:53.647028 1559 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d5d3dbb83324641a6fc9c439b1bd217 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:43:53.647208 1559 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d5d3dbb83324641a6fc9c439b1bd217 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:43:53.647410 1559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d5d3dbb83324641a6fc9c439b1bd217 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:43:53.650255 1559 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d5d3dbb83324641a6fc9c439b1bd217 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5d3dbb83324641a6fc9c439b1bd217" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35969 } }
I20260928 08:43:53.651078 1559 leader_election.cc:302] T 00000000000000000000000000000000 P 4d5d3dbb83324641a6fc9c439b1bd217 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d5d3dbb83324641a6fc9c439b1bd217; no voters:
I20260928 08:43:53.651947 1559 leader_election.cc:288] T 00000000000000000000000000000000 P 4d5d3dbb83324641a6fc9c439b1bd217 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:43:53.652100 1564 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d5d3dbb83324641a6fc9c439b1bd217 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:43:53.652871 1564 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d5d3dbb83324641a6fc9c439b1bd217 [term 1 LEADER]: Becoming Leader. State: Replica: 4d5d3dbb83324641a6fc9c439b1bd217, State: Running, Role: LEADER
I20260928 08:43:53.653540 1564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d5d3dbb83324641a6fc9c439b1bd217 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5d3dbb83324641a6fc9c439b1bd217" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35969 } }
I20260928 08:43:53.654943 1559 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d5d3dbb83324641a6fc9c439b1bd217 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:43:53.656648 1566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d5d3dbb83324641a6fc9c439b1bd217 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d5d3dbb83324641a6fc9c439b1bd217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5d3dbb83324641a6fc9c439b1bd217" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35969 } } }
I20260928 08:43:53.657058 1566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d5d3dbb83324641a6fc9c439b1bd217 [sys.catalog]: This master's current role is: LEADER
I20260928 08:43:53.657783 1565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d5d3dbb83324641a6fc9c439b1bd217 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d5d3dbb83324641a6fc9c439b1bd217. Latest consensus state: current_term: 1 leader_uuid: "4d5d3dbb83324641a6fc9c439b1bd217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d5d3dbb83324641a6fc9c439b1bd217" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35969 } } }
I20260928 08:43:53.658087 1571 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:43:53.658128 1565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d5d3dbb83324641a6fc9c439b1bd217 [sys.catalog]: This master's current role is: LEADER
I20260928 08:43:53.664932 1571 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:43:53.674991 1571 catalog_manager.cc:1347] Generated new cluster ID: 5079835fa37c4f0a995547f2050e42d7
I20260928 08:43:53.675118 1571 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:43:53.704236 1571 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:43:53.705601 1571 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:43:53.713240 1571 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d5d3dbb83324641a6fc9c439b1bd217: Generated new TSK 0
I20260928 08:43:53.714437 1571 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:43:53.768503 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35969
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:43:54.091086 1585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:43:54.091416 1585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:43:54.091603 1585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:43:54.098589 1585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:43:54.098737 1585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:43:54.098905 1585 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:43:54.099051 1585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:43:54.108381 1585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:35969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:43:54.109643 1585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:43:54.110976 1585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:43:54.124483 1590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:54.124394 1593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:54.124449 1591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:43:54.125403 1585 server_base.cc:1034] running on GCE node
I20260928 08:43:54.126019 1585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:43:54.126931 1585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:43:54.128166 1585 hybrid_clock.cc:648] HybridClock initialized: now 1790585034128104 us; error 42 us; skew 500 ppm
I20260928 08:43:54.128472 1585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:43:54 dist-test-slave-n6xs krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:43:54.131631 1585 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:43:54.133096 1585 webserver.cc:460] Webserver started at http://127.0.97.1:34693/ using document root <none> and password file <none>
I20260928 08:43:54.133713 1585 fs_manager.cc:359] Metadata directory not provided
I20260928 08:43:54.133798 1585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:43:54.134057 1585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:43:54.137054 1585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-0/data/instance:
uuid: "2c97af2a0a5a448c99b40aaf2e1bd520"
format_stamp: "Formatted at 2026-09-28 08:43:54 on dist-test-slave-n6xs"
server_key: "5aa3aa516723e8491e8a21cc7ee6762e"
server_key_iv: "a778505626f122652495579257652007"
server_key_version: "encryptionkey@0"
I20260928 08:43:54.137765 1585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-0/wal/instance:
uuid: "2c97af2a0a5a448c99b40aaf2e1bd520"
format_stamp: "Formatted at 2026-09-28 08:43:54 on dist-test-slave-n6xs"
server_key: "5aa3aa516723e8491e8a21cc7ee6762e"
server_key_iv: "a778505626f122652495579257652007"
server_key_version: "encryptionkey@0"
I20260928 08:43:54.142673 1585 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:43:54.146075 1600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:43:54.147445 1585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:43:54.147565 1585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-0/wal
uuid: "2c97af2a0a5a448c99b40aaf2e1bd520"
format_stamp: "Formatted at 2026-09-28 08:43:54 on dist-test-slave-n6xs"
server_key: "5aa3aa516723e8491e8a21cc7ee6762e"
server_key_iv: "a778505626f122652495579257652007"
server_key_version: "encryptionkey@0"
I20260928 08:43:54.147738 1585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:43:54.176136 1585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:43:54.180135 1585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:43:54.180347 1585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:43:54.182808 1585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:43:54.182915 1585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:43:54.183043 1585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:43:54.183117 1585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:43:54 dist-test-slave-n6xs krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:43:54.207217 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:54.183533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:43249 (local address 127.0.97.62:35969)
0928 08:43:54.184051 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:54.184055 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:54.185192 (+ 1137us) server_negotiation.cc:407] Connection header received
0928 08:43:54.187173 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:54.187179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:54.187271 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:54.187414 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:54.190062 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:54.190780 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:54.193932 (+ 3152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:54.194537 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:54.194675 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:54.199991 (+ 5316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:54.200000 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:54.200007 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:54.200053 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:54.203299 (+ 3246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:54.204262 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:54.204269 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:54.204275 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:54.204426 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:54.205020 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:54.205029 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:54.205035 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:54.205530 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:54.205716 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:54.206455 (+ 739us) server_negotiation.cc:299] Negotiation successful
0928 08:43:54.206744 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":300,"threads_started":1}
I20260928 08:43:54.208391 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:54.184188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35969 (local address 127.0.97.1:43249)
0928 08:43:54.185030 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:54.185074 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:54.186906 (+ 1832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:54.187735 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:54.187752 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:54.188709 (+ 957us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:43:54.189846 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:54.189878 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:54.190925 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:54.190933 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:54.193761 (+ 2828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:54.193779 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:54.194823 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:54.194833 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:54.195099 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:54.196000 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:43:54.196056 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:43:54.199681 (+ 3625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:43:54.203431 (+ 3750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:54.203443 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:54.203459 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:54.204068 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:54.204548 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:54.204559 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:54.204561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:54.204869 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:54.205700 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:43:54.205706 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:43:54.206074 (+ 368us) client_negotiation.cc:769] Sending connection context
0928 08:43:54.206298 (+ 224us) client_negotiation.cc:240] Negotiation successful
0928 08:43:54.206667 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":386,"threads_started":1}
I20260928 08:43:54.233243 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:54.218363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60575 (local address 127.0.97.62:35969)
0928 08:43:54.218583 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:54.218590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:54.219244 (+ 654us) server_negotiation.cc:407] Connection header received
0928 08:43:54.219444 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:54.219448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:54.219508 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:54.219584 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:54.221177 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:54.222055 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:54.224218 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:54.224775 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:54.224923 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:54.227214 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:54.227218 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:54.227221 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:54.227268 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:54.230343 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:54.231202 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:54.231210 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:54.231214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:54.231365 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:54.231832 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:54.231839 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:54.231844 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:54.232158 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:54.232306 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:54.232685 (+ 379us) server_negotiation.cc:299] Negotiation successful
0928 08:43:54.232898 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 08:43:54.233345 1679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:54.218451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35969 (local address 127.0.97.1:60575)
0928 08:43:54.219122 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:54.219147 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:54.219314 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:54.219648 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:54.219655 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:54.220447 (+ 792us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:43:54.220987 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:54.221005 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:54.222252 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:54.222260 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:54.224008 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:54.224020 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:54.225157 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:54.225162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:54.225337 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:54.226090 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:43:54.226106 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:43:54.226985 (+ 879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:43:54.230475 (+ 3490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:54.230480 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:54.230484 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:54.231098 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:54.231485 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:54.231493 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:54.231499 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:54.231675 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:54.232281 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:43:54.232285 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:43:54.232464 (+ 179us) client_negotiation.cc:769] Sending connection context
0928 08:43:54.232598 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 08:43:54.232905 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":315,"threads_started":1}
I20260928 08:43:54.238945 1585 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39109
I20260928 08:43:54.239038 1720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39109 every 8 connection(s)
I20260928 08:43:54.240181 1585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:43:54.242303 1721 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35969
I20260928 08:43:54.242514 1721 heartbeater.cc:464] Registering TS with master...
I20260928 08:43:54.243105 1721 heartbeater.cc:511] Master 127.0.97.62:35969 requested a full tablet report, sending...
I20260928 08:43:54.245085 1514 ts_manager.cc:194] Registered new tserver with Master: 2c97af2a0a5a448c99b40aaf2e1bd520 (127.0.97.1:39109)
I20260928 08:43:54.246555 1514 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:43249
I20260928 08:43:54.247714 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 1585
I20260928 08:43:54.247929 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:43:54.248559 388 external_mini_cluster.cc:1428] Setting key 7089807b4d09c26334a00be654cc5c04
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 08:43:54.289004 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35969
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:43:54.599937 1725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:43:54.600260 1725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:43:54.600425 1725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:43:54.607043 1725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:43:54.607187 1725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:43:54.607336 1725 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:43:54.607501 1725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:43:54.617185 1725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:35969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:43:54.618424 1725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:43:54.619701 1725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:43:54.632557 1730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:54.632589 1733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:54.632665 1731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:43:54.633867 1725 server_base.cc:1034] running on GCE node
I20260928 08:43:54.634488 1725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:43:54.635423 1725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:43:54.636691 1725 hybrid_clock.cc:648] HybridClock initialized: now 1790585034636622 us; error 54 us; skew 500 ppm
I20260928 08:43:54.636986 1725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:43:54 dist-test-slave-n6xs krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:43:54.647704 1725 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:43:54.649205 1725 webserver.cc:460] Webserver started at http://127.0.97.2:38515/ using document root <none> and password file <none>
I20260928 08:43:54.649863 1725 fs_manager.cc:359] Metadata directory not provided
I20260928 08:43:54.649998 1725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:43:54.650310 1725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:43:54.653363 1725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-1/data/instance:
uuid: "4c5946ff59fc46d7a89d744559e88e9a"
format_stamp: "Formatted at 2026-09-28 08:43:54 on dist-test-slave-n6xs"
server_key: "aa3f2d98a3f8b968f6c5243c3a978702"
server_key_iv: "6bc070d727d8a773e134c444448345b1"
server_key_version: "encryptionkey@0"
I20260928 08:43:54.654146 1725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-1/wal/instance:
uuid: "4c5946ff59fc46d7a89d744559e88e9a"
format_stamp: "Formatted at 2026-09-28 08:43:54 on dist-test-slave-n6xs"
server_key: "aa3f2d98a3f8b968f6c5243c3a978702"
server_key_iv: "6bc070d727d8a773e134c444448345b1"
server_key_version: "encryptionkey@0"
I20260928 08:43:54.659138 1725 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 08:43:54.662530 1740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:43:54.663729 1725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:43:54.663916 1725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-1/wal
uuid: "4c5946ff59fc46d7a89d744559e88e9a"
format_stamp: "Formatted at 2026-09-28 08:43:54 on dist-test-slave-n6xs"
server_key: "aa3f2d98a3f8b968f6c5243c3a978702"
server_key_iv: "6bc070d727d8a773e134c444448345b1"
server_key_version: "encryptionkey@0"
I20260928 08:43:54.664160 1725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:43:54.675755 1725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:43:54.679425 1725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:43:54.679690 1725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:43:54.682209 1725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:43:54.682313 1725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:43:54.682492 1725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:43:54.682570 1725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:43:54 dist-test-slave-n6xs krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:43:54.704176 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:54.682521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47325 (local address 127.0.97.62:35969)
0928 08:43:54.682724 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:54.682731 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:54.684132 (+ 1401us) server_negotiation.cc:407] Connection header received
0928 08:43:54.686046 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:54.686052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:54.686137 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:54.686242 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:54.688429 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:54.689527 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:54.691068 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:54.691441 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:54.691569 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:54.695916 (+ 4347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:54.695921 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:54.695924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:54.695954 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:54.699539 (+ 3585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:54.700661 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:54.700682 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:54.700687 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:54.700851 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:54.701336 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:54.701341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:54.701344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:54.701739 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:54.701888 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:54.703732 (+ 1844us) server_negotiation.cc:299] Negotiation successful
0928 08:43:54.703967 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 08:43:54.704485 1749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:54.683084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35969 (local address 127.0.97.2:47325)
0928 08:43:54.683969 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:54.684011 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:54.685741 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:54.686476 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:54.686485 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:54.687216 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:43:54.688191 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:54.688221 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:54.689673 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:54.689680 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:54.690917 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:54.690927 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:54.691667 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:54.691673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:54.691959 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:54.692749 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:43:54.692784 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:43:54.695627 (+ 2843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:43:54.699649 (+ 4022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:54.699657 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:54.699671 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:54.700499 (+ 828us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:54.700994 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:54.701000 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:54.701006 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:54.701217 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:54.701852 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:43:54.701860 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:43:54.702319 (+ 459us) client_negotiation.cc:769] Sending connection context
0928 08:43:54.702604 (+ 285us) client_negotiation.cc:240] Negotiation successful
0928 08:43:54.703047 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":350,"threads_started":1}
I20260928 08:43:54.728053 1725 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46517
I20260928 08:43:54.728106 1859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46517 every 8 connection(s)
I20260928 08:43:54.729442 1725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:43:54.731640 1860 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35969
I20260928 08:43:54.731887 1860 heartbeater.cc:464] Registering TS with master...
I20260928 08:43:54.732587 1860 heartbeater.cc:511] Master 127.0.97.62:35969 requested a full tablet report, sending...
I20260928 08:43:54.732827 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:54.718774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49713 (local address 127.0.97.62:35969)
0928 08:43:54.719005 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:54.719014 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:54.720057 (+ 1043us) server_negotiation.cc:407] Connection header received
0928 08:43:54.720330 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:54.720337 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:54.720423 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:54.720531 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:54.722064 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:54.723065 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:54.724955 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:54.725377 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:54.725523 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:54.727170 (+ 1647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:54.727177 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:54.727182 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:54.727225 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:54.729859 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:54.730697 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:54.730704 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:54.730708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:54.730827 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:54.731277 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:54.731289 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:54.731291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:54.731687 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:54.731873 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:54.732296 (+ 423us) server_negotiation.cc:299] Negotiation successful
0928 08:43:54.732496 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 08:43:54.733006 1838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:54.718745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35969 (local address 127.0.97.2:49713)
0928 08:43:54.719922 (+ 1177us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:54.719957 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:54.720125 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:54.720682 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:54.720685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:54.721334 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:43:54.721929 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:54.721939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:54.723210 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:54.723215 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:54.724811 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:54.724825 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:54.725568 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:54.725574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:54.725732 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:54.726325 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:43:54.726341 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:43:54.726992 (+ 651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:43:54.729999 (+ 3007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:54.730004 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:54.730008 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:54.730565 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:54.730976 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:54.730981 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:54.730985 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:54.731151 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:54.731857 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:43:54.731861 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:43:54.732059 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 08:43:54.732204 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 08:43:54.732528 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":970,"thread_start_us":780,"threads_started":1}
I20260928 08:43:54.734076 1513 ts_manager.cc:194] Registered new tserver with Master: 4c5946ff59fc46d7a89d744559e88e9a (127.0.97.2:46517)
I20260928 08:43:54.734920 1513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:47325
I20260928 08:43:54.735196 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 1725
I20260928 08:43:54.735360 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:43:54.735769 388 external_mini_cluster.cc:1428] Setting key 801507b289d29342dcef0e1610bdad28
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:43:54.772413 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35969
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:43:55.085260 1864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:43:55.085536 1864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:43:55.085685 1864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:43:55.093101 1864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:43:55.093204 1864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:43:55.093348 1864 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:43:55.093466 1864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:43:55.103398 1864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:35969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:43:55.104753 1864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:43:55.106251 1864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:43:55.120368 1870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:55.120718 1869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:55.120857 1872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:43:55.121359 1864 server_base.cc:1034] running on GCE node
I20260928 08:43:55.121924 1864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:43:55.122798 1864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:43:55.123996 1864 hybrid_clock.cc:648] HybridClock initialized: now 1790585035123942 us; error 38 us; skew 500 ppm
I20260928 08:43:55.124300 1864 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:43:55 dist-test-slave-n6xs krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:43:55.127240 1864 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:43:55.128767 1864 webserver.cc:460] Webserver started at http://127.0.97.3:36089/ using document root <none> and password file <none>
I20260928 08:43:55.129419 1864 fs_manager.cc:359] Metadata directory not provided
I20260928 08:43:55.129546 1864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:43:55.129874 1864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:43:55.132627 1864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-2/data/instance:
uuid: "cab11df9b0724ca7ba16a8938196064c"
format_stamp: "Formatted at 2026-09-28 08:43:55 on dist-test-slave-n6xs"
server_key: "a4878366ccf0201519809abebc69c920"
server_key_iv: "77f940b851f676d9324ffb17d66989cb"
server_key_version: "encryptionkey@0"
I20260928 08:43:55.133369 1864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-2/wal/instance:
uuid: "cab11df9b0724ca7ba16a8938196064c"
format_stamp: "Formatted at 2026-09-28 08:43:55 on dist-test-slave-n6xs"
server_key: "a4878366ccf0201519809abebc69c920"
server_key_iv: "77f940b851f676d9324ffb17d66989cb"
server_key_version: "encryptionkey@0"
I20260928 08:43:55.138286 1864 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:43:55.141713 1879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:43:55.143056 1864 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260928 08:43:55.143222 1864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-2/wal
uuid: "cab11df9b0724ca7ba16a8938196064c"
format_stamp: "Formatted at 2026-09-28 08:43:55 on dist-test-slave-n6xs"
server_key: "a4878366ccf0201519809abebc69c920"
server_key_iv: "77f940b851f676d9324ffb17d66989cb"
server_key_version: "encryptionkey@0"
I20260928 08:43:55.143424 1864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:43:55.157415 1864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:43:55.161413 1864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:43:55.161686 1864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:43:55.164376 1864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:43:55.164472 1864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:43:55.164635 1864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:43:55.164757 1864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 28 08:43:55 dist-test-slave-n6xs krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:43:55.187613 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:55.164919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50611 (local address 127.0.97.62:35969)
0928 08:43:55.165137 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:55.165146 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:55.166508 (+ 1362us) server_negotiation.cc:407] Connection header received
0928 08:43:55.168886 (+ 2378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:55.168892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:55.168971 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:55.169094 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:55.171645 (+ 2551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.173008 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.175451 (+ 2443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.176215 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.176385 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:55.181120 (+ 4735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:55.181127 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:55.181131 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:55.181211 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:55.183873 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:55.184796 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:55.184803 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:55.184813 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:55.184932 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:55.185504 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:55.185508 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:55.185510 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:55.185873 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:55.186044 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:55.187110 (+ 1066us) server_negotiation.cc:299] Negotiation successful
0928 08:43:55.187369 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 08:43:55.189656 1888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:55.165487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35969 (local address 127.0.97.3:50611)
0928 08:43:55.166340 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:55.166391 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:55.168545 (+ 2154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:55.169341 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:55.169356 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:55.170121 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:43:55.171422 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:55.171453 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.173229 (+ 1776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.173240 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:55.175302 (+ 2062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:55.175320 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.176497 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.176510 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:55.176729 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:55.177530 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:43:55.177567 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:43:55.180907 (+ 3340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:43:55.184016 (+ 3109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:55.184024 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:55.184042 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:55.184663 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:55.185058 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:55.185063 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:55.185066 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:55.185330 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:55.186066 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:43:55.186079 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:43:55.186636 (+ 557us) client_negotiation.cc:769] Sending connection context
0928 08:43:55.187035 (+ 399us) client_negotiation.cc:240] Negotiation successful
0928 08:43:55.187749 (+ 714us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":364,"threads_started":1}
I20260928 08:43:55.209262 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:55.196225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40839 (local address 127.0.97.62:35969)
0928 08:43:55.196392 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:55.196396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:55.197280 (+ 884us) server_negotiation.cc:407] Connection header received
0928 08:43:55.197491 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:55.197495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:55.197555 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:55.197656 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:55.198515 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.199225 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.200653 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.201041 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.201165 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:55.202853 (+ 1688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:55.202858 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:55.202861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:55.202889 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:55.206021 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:55.206712 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:55.206719 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:55.206723 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:55.206885 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:55.207423 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:55.207431 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:55.207437 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:55.207963 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:55.208409 (+ 446us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:55.208697 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 08:43:55.208904 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 08:43:55.209321 1952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:55.196313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35969 (local address 127.0.97.3:40839)
0928 08:43:55.197181 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:55.197198 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:55.197367 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:55.197676 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:55.197680 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:55.198108 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:43:55.198375 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:55.198383 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.199363 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.199368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:55.200476 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:55.200489 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.201227 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.201233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:55.201387 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:55.201991 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:43:55.202009 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:43:55.202665 (+ 656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:43:55.206158 (+ 3493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:55.206163 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:55.206171 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:55.206596 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:55.207017 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:55.207038 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:55.207044 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:55.207234 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:55.208125 (+ 891us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:43:55.208130 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:43:55.208432 (+ 302us) client_negotiation.cc:769] Sending connection context
0928 08:43:55.208593 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 08:43:55.208904 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":504,"threads_started":1}
I20260928 08:43:55.220671 1864 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37435
I20260928 08:43:55.220693 1998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37435 every 8 connection(s)
I20260928 08:43:55.221904 1864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:43:55.223930 1999 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35969
I20260928 08:43:55.224159 1999 heartbeater.cc:464] Registering TS with master...
I20260928 08:43:55.224810 1999 heartbeater.cc:511] Master 127.0.97.62:35969 requested a full tablet report, sending...
I20260928 08:43:55.226145 1513 ts_manager.cc:194] Registered new tserver with Master: cab11df9b0724ca7ba16a8938196064c (127.0.97.3:37435)
I20260928 08:43:55.226905 1513 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50611
I20260928 08:43:55.228196 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 1864
I20260928 08:43:55.228370 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:43:55.228874 388 external_mini_cluster.cc:1428] Setting key 8eada94ce6da0a3f33aab0949643e30a
I20260928 08:43:55.231724 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:43:55.249526 1721 heartbeater.cc:503] Master 127.0.97.62:35969 was elected leader, sending a full tablet report...
Sep 28 08:43:55 dist-test-slave-n6xs krb5kdc[1469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585035, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 08:43:55 dist-test-slave-n6xs krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585035, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:43:55.276801 1609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:55.263117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49680 (local address 127.0.97.62:35969)
0928 08:43:55.263368 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:55.263375 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:55.263611 (+ 236us) server_negotiation.cc:407] Connection header received
0928 08:43:55.263840 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:55.263848 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:55.263924 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:55.264032 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:55.265413 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.266154 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.267505 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.268065 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.268198 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:55.271875 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:55.271886 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:55.271889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:55.271933 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:55.274220 (+ 2287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:55.274821 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:55.274828 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:55.274836 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:55.274947 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:55.275348 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:55.275359 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:55.275362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:55.275606 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:55.275871 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:55.276187 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 08:43:55.276424 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 08:43:55.280874 1513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:43:55.287600 1513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:43:55.315150 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:55.299670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44362 (local address 127.0.97.3:37435)
0928 08:43:55.301519 (+ 1849us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:55.301529 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:55.302382 (+ 853us) server_negotiation.cc:407] Connection header received
0928 08:43:55.302707 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:55.302714 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:55.303019 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:55.303239 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:43:55.304536 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.306782 (+ 2246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.310606 (+ 3824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.312865 (+ 2259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.313016 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:55.313257 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:55.314679 (+ 1422us) server_negotiation.cc:299] Negotiation successful
0928 08:43:55.314852 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1683,"thread_start_us":1527,"threads_started":1}
I20260928 08:43:55.315128 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:55.299417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37435 (local address 127.0.0.1:44362)
0928 08:43:55.301972 (+ 2555us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:55.302009 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:55.302442 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:55.303618 (+ 1176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:55.303629 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:55.303671 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:43:55.304246 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:55.304256 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.306982 (+ 2726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.306991 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:55.310299 (+ 3308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:55.310320 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.314133 (+ 3813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.314141 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:55.314382 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:55.314442 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 08:43:55.314569 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 08:43:55.314752 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2272,"thread_start_us":2099,"threads_started":1}
I20260928 08:43:55.319643 1929 tablet_service.cc:1467] Processing CreateTablet for tablet 236b5aff0aef48249fc312c9a8b3d1ce (DEFAULT_TABLE table=test-table [id=98dd05032e9b4cb69541649c5091aa30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:43:55.321347 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 236b5aff0aef48249fc312c9a8b3d1ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:43:55.322146 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:55.300118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42078 (local address 127.0.97.2:46517)
0928 08:43:55.300965 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:55.300978 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:55.313535 (+ 12557us) server_negotiation.cc:407] Connection header received
0928 08:43:55.314677 (+ 1142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:55.314686 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:55.314979 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:55.315185 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:43:55.316270 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.317630 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.319545 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.320803 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.320933 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:55.321123 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:55.321659 (+ 536us) server_negotiation.cc:299] Negotiation successful
0928 08:43:55.321832 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":406,"threads_started":1}
I20260928 08:43:55.322885 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:55.299496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46517 (local address 127.0.0.1:42078)
0928 08:43:55.313361 (+ 13865us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:55.313387 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:55.314473 (+ 1086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:55.315394 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:55.315399 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:55.315442 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:43:55.316112 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:55.316125 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.317773 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.317780 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:55.319388 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:55.319400 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.321260 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.321268 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:55.321453 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:55.321486 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 08:43:55.322415 (+ 929us) client_negotiation.cc:240] Negotiation successful
0928 08:43:55.322532 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13682,"mutex_wait_us":112,"spinlock_wait_cycles":2944,"thread_start_us":14725,"threads_started":1}
I20260928 08:43:55.322916 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:55.302315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39109 (local address 127.0.0.1:47908)
0928 08:43:55.314515 (+ 12200us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:55.314528 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:55.314671 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:55.315394 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:55.315399 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:55.315446 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:43:55.316037 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:55.316056 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.317808 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.317813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:55.320011 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:55.320024 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.322042 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.322050 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:55.322210 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:55.322241 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 08:43:55.322318 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 08:43:55.322459 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12071,"mutex_wait_us":97,"spinlock_wait_cycles":5760,"thread_start_us":10664,"threads_started":1}
I20260928 08:43:55.323325 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:55.302580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47908 (local address 127.0.97.1:39109)
0928 08:43:55.314018 (+ 11438us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:55.314030 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:55.314780 (+ 750us) server_negotiation.cc:407] Connection header received
0928 08:43:55.314872 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:55.314877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:55.315059 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:55.315205 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:43:55.316235 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.317640 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.320163 (+ 2523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.321849 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.321978 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:55.322164 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:55.322903 (+ 739us) server_negotiation.cc:299] Negotiation successful
0928 08:43:55.323063 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11235,"thread_start_us":11593,"threads_started":1}
I20260928 08:43:55.326278 1651 tablet_service.cc:1467] Processing CreateTablet for tablet 236b5aff0aef48249fc312c9a8b3d1ce (DEFAULT_TABLE table=test-table [id=98dd05032e9b4cb69541649c5091aa30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:43:55.326505 1790 tablet_service.cc:1467] Processing CreateTablet for tablet 236b5aff0aef48249fc312c9a8b3d1ce (DEFAULT_TABLE table=test-table [id=98dd05032e9b4cb69541649c5091aa30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:43:55.327482 1651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 236b5aff0aef48249fc312c9a8b3d1ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:43:55.327746 1790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 236b5aff0aef48249fc312c9a8b3d1ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:43:55.335791 2018 tablet_bootstrap.cc:492] T 236b5aff0aef48249fc312c9a8b3d1ce P cab11df9b0724ca7ba16a8938196064c: Bootstrap starting.
I20260928 08:43:55.339362 2019 tablet_bootstrap.cc:492] T 236b5aff0aef48249fc312c9a8b3d1ce P 2c97af2a0a5a448c99b40aaf2e1bd520: Bootstrap starting.
I20260928 08:43:55.339545 2018 tablet_bootstrap.cc:654] T 236b5aff0aef48249fc312c9a8b3d1ce P cab11df9b0724ca7ba16a8938196064c: Neither blocks nor log segments found. Creating new log.
I20260928 08:43:55.341073 2018 log.cc:824] T 236b5aff0aef48249fc312c9a8b3d1ce P cab11df9b0724ca7ba16a8938196064c: Log is configured to *not* fsync() on all Append() calls
I20260928 08:43:55.341344 2020 tablet_bootstrap.cc:492] T 236b5aff0aef48249fc312c9a8b3d1ce P 4c5946ff59fc46d7a89d744559e88e9a: Bootstrap starting.
I20260928 08:43:55.342566 2019 tablet_bootstrap.cc:654] T 236b5aff0aef48249fc312c9a8b3d1ce P 2c97af2a0a5a448c99b40aaf2e1bd520: Neither blocks nor log segments found. Creating new log.
I20260928 08:43:55.343479 2019 log.cc:824] T 236b5aff0aef48249fc312c9a8b3d1ce P 2c97af2a0a5a448c99b40aaf2e1bd520: Log is configured to *not* fsync() on all Append() calls
I20260928 08:43:55.345650 2020 tablet_bootstrap.cc:654] T 236b5aff0aef48249fc312c9a8b3d1ce P 4c5946ff59fc46d7a89d744559e88e9a: Neither blocks nor log segments found. Creating new log.
I20260928 08:43:55.346529 2019 tablet_bootstrap.cc:492] T 236b5aff0aef48249fc312c9a8b3d1ce P 2c97af2a0a5a448c99b40aaf2e1bd520: No bootstrap required, opened a new log
I20260928 08:43:55.346817 2019 ts_tablet_manager.cc:1397] T 236b5aff0aef48249fc312c9a8b3d1ce P 2c97af2a0a5a448c99b40aaf2e1bd520: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260928 08:43:55.347111 2020 log.cc:824] T 236b5aff0aef48249fc312c9a8b3d1ce P 4c5946ff59fc46d7a89d744559e88e9a: Log is configured to *not* fsync() on all Append() calls
I20260928 08:43:55.349668 2020 tablet_bootstrap.cc:492] T 236b5aff0aef48249fc312c9a8b3d1ce P 4c5946ff59fc46d7a89d744559e88e9a: No bootstrap required, opened a new log
I20260928 08:43:55.349934 2020 ts_tablet_manager.cc:1397] T 236b5aff0aef48249fc312c9a8b3d1ce P 4c5946ff59fc46d7a89d744559e88e9a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 08:43:55.350162 2018 tablet_bootstrap.cc:492] T 236b5aff0aef48249fc312c9a8b3d1ce P cab11df9b0724ca7ba16a8938196064c: No bootstrap required, opened a new log
I20260928 08:43:55.350410 2018 ts_tablet_manager.cc:1397] T 236b5aff0aef48249fc312c9a8b3d1ce P cab11df9b0724ca7ba16a8938196064c: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20260928 08:43:55.357684 2020 raft_consensus.cc:348] T 236b5aff0aef48249fc312c9a8b3d1ce P 4c5946ff59fc46d7a89d744559e88e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } }
I20260928 08:43:55.358044 2020 raft_consensus.cc:374] T 236b5aff0aef48249fc312c9a8b3d1ce P 4c5946ff59fc46d7a89d744559e88e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:43:55.357873 2019 raft_consensus.cc:348] T 236b5aff0aef48249fc312c9a8b3d1ce P 2c97af2a0a5a448c99b40aaf2e1bd520 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } }
I20260928 08:43:55.358143 2020 raft_consensus.cc:729] T 236b5aff0aef48249fc312c9a8b3d1ce P 4c5946ff59fc46d7a89d744559e88e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c5946ff59fc46d7a89d744559e88e9a, State: Initialized, Role: FOLLOWER
I20260928 08:43:55.358273 2019 raft_consensus.cc:374] T 236b5aff0aef48249fc312c9a8b3d1ce P 2c97af2a0a5a448c99b40aaf2e1bd520 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:43:55.358361 2019 raft_consensus.cc:729] T 236b5aff0aef48249fc312c9a8b3d1ce P 2c97af2a0a5a448c99b40aaf2e1bd520 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c97af2a0a5a448c99b40aaf2e1bd520, State: Initialized, Role: FOLLOWER
I20260928 08:43:55.358896 2019 consensus_queue.cc:260] T 236b5aff0aef48249fc312c9a8b3d1ce P 2c97af2a0a5a448c99b40aaf2e1bd520 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } }
I20260928 08:43:55.359295 2020 consensus_queue.cc:260] T 236b5aff0aef48249fc312c9a8b3d1ce P 4c5946ff59fc46d7a89d744559e88e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } }
I20260928 08:43:55.360440 1860 heartbeater.cc:503] Master 127.0.97.62:35969 was elected leader, sending a full tablet report...
I20260928 08:43:55.360983 2020 ts_tablet_manager.cc:1428] T 236b5aff0aef48249fc312c9a8b3d1ce P 4c5946ff59fc46d7a89d744559e88e9a: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260928 08:43:55.361801 2018 raft_consensus.cc:348] T 236b5aff0aef48249fc312c9a8b3d1ce P cab11df9b0724ca7ba16a8938196064c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } }
I20260928 08:43:55.362222 2018 raft_consensus.cc:374] T 236b5aff0aef48249fc312c9a8b3d1ce P cab11df9b0724ca7ba16a8938196064c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:43:55.362332 2018 raft_consensus.cc:729] T 236b5aff0aef48249fc312c9a8b3d1ce P cab11df9b0724ca7ba16a8938196064c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab11df9b0724ca7ba16a8938196064c, State: Initialized, Role: FOLLOWER
I20260928 08:43:55.362607 2019 ts_tablet_manager.cc:1428] T 236b5aff0aef48249fc312c9a8b3d1ce P 2c97af2a0a5a448c99b40aaf2e1bd520: Time spent starting tablet: real 0.016s user 0.007s sys 0.007s
I20260928 08:43:55.362972 2018 consensus_queue.cc:260] T 236b5aff0aef48249fc312c9a8b3d1ce P cab11df9b0724ca7ba16a8938196064c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } }
I20260928 08:43:55.364048 1999 heartbeater.cc:503] Master 127.0.97.62:35969 was elected leader, sending a full tablet report...
I20260928 08:43:55.364616 2018 ts_tablet_manager.cc:1428] T 236b5aff0aef48249fc312c9a8b3d1ce P cab11df9b0724ca7ba16a8938196064c: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
W20260928 08:43:55.473527 2000 tablet.cc:2369] T 236b5aff0aef48249fc312c9a8b3d1ce P cab11df9b0724ca7ba16a8938196064c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:43:55.481982 1861 tablet.cc:2369] T 236b5aff0aef48249fc312c9a8b3d1ce P 4c5946ff59fc46d7a89d744559e88e9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:43:55.492489 1722 tablet.cc:2369] T 236b5aff0aef48249fc312c9a8b3d1ce P 2c97af2a0a5a448c99b40aaf2e1bd520: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:43:55.620874 2024 raft_consensus.cc:482] T 236b5aff0aef48249fc312c9a8b3d1ce P 2c97af2a0a5a448c99b40aaf2e1bd520 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:43:55.621204 2024 raft_consensus.cc:504] T 236b5aff0aef48249fc312c9a8b3d1ce P 2c97af2a0a5a448c99b40aaf2e1bd520 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } }
I20260928 08:43:55.622897 2024 leader_election.cc:288] T 236b5aff0aef48249fc312c9a8b3d1ce P 2c97af2a0a5a448c99b40aaf2e1bd520 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cab11df9b0724ca7ba16a8938196064c (127.0.97.3:37435), 4c5946ff59fc46d7a89d744559e88e9a (127.0.97.2:46517)
I20260928 08:43:55.631894 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:55.623395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34983 (local address 127.0.97.3:37435)
0928 08:43:55.623564 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:55.623570 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:55.624303 (+ 733us) server_negotiation.cc:407] Connection header received
0928 08:43:55.624495 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:55.624500 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:55.624569 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:55.624675 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:43:55.625591 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.626690 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.629426 (+ 2736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.630754 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.630906 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:55.631040 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:55.631412 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 08:43:55.631554 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 08:43:55.631933 2015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:55.623302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58581 (local address 127.0.97.2:46517)
0928 08:43:55.623500 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:55.623507 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:55.624302 (+ 795us) server_negotiation.cc:407] Connection header received
0928 08:43:55.624481 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:55.624486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:55.624566 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:55.624674 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:43:55.625589 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.626705 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.629624 (+ 2919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.630922 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.631059 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:55.631165 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:55.631423 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 08:43:55.631559 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 08:43:55.632504 2027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:55.623207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37435 (local address 127.0.97.1:34983)
0928 08:43:55.624159 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:55.624179 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:55.624345 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:55.624744 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:55.624753 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:55.624791 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:43:55.625417 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:55.625427 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.626861 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.626887 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:55.629233 (+ 2346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:55.629248 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.630981 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.631012 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:55.631214 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:55.631252 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 08:43:55.631362 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 08:43:55.631564 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"mutex_wait_us":106,"spinlock_wait_cycles":52864,"thread_start_us":444,"threads_started":1}
I20260928 08:43:55.632688 2028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:55.623207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46517 (local address 127.0.97.1:58581)
0928 08:43:55.624106 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:55.624125 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:55.624303 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:55.624744 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:55.624749 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:55.624791 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:43:55.625376 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:55.625393 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.627350 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.627355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:55.629435 (+ 2080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:55.629455 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.631126 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.631134 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:55.631272 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:55.631309 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 08:43:55.631385 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 08:43:55.631564 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":739,"mutex_wait_us":63,"spinlock_wait_cycles":1792,"thread_start_us":536,"threads_started":1}
I20260928 08:43:55.633626 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "236b5aff0aef48249fc312c9a8b3d1ce" candidate_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5946ff59fc46d7a89d744559e88e9a" is_pre_election: true
I20260928 08:43:55.633625 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "236b5aff0aef48249fc312c9a8b3d1ce" candidate_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab11df9b0724ca7ba16a8938196064c" is_pre_election: true
I20260928 08:43:55.634110 1811 raft_consensus.cc:2454] T 236b5aff0aef48249fc312c9a8b3d1ce P 4c5946ff59fc46d7a89d744559e88e9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c97af2a0a5a448c99b40aaf2e1bd520 in term 0.
I20260928 08:43:55.634124 1954 raft_consensus.cc:2454] T 236b5aff0aef48249fc312c9a8b3d1ce P cab11df9b0724ca7ba16a8938196064c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c97af2a0a5a448c99b40aaf2e1bd520 in term 0.
I20260928 08:43:55.634788 1603 leader_election.cc:302] T 236b5aff0aef48249fc312c9a8b3d1ce P 2c97af2a0a5a448c99b40aaf2e1bd520 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2c97af2a0a5a448c99b40aaf2e1bd520, 4c5946ff59fc46d7a89d744559e88e9a, cab11df9b0724ca7ba16a8938196064c; no voters:
I20260928 08:43:55.635182 2024 raft_consensus.cc:2781] T 236b5aff0aef48249fc312c9a8b3d1ce P 2c97af2a0a5a448c99b40aaf2e1bd520 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:43:55.635335 2024 raft_consensus.cc:482] T 236b5aff0aef48249fc312c9a8b3d1ce P 2c97af2a0a5a448c99b40aaf2e1bd520 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:43:55.635440 2024 raft_consensus.cc:3037] T 236b5aff0aef48249fc312c9a8b3d1ce P 2c97af2a0a5a448c99b40aaf2e1bd520 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:43:55.637677 2024 raft_consensus.cc:504] T 236b5aff0aef48249fc312c9a8b3d1ce P 2c97af2a0a5a448c99b40aaf2e1bd520 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } }
I20260928 08:43:55.638283 2024 leader_election.cc:288] T 236b5aff0aef48249fc312c9a8b3d1ce P 2c97af2a0a5a448c99b40aaf2e1bd520 [CANDIDATE]: Term 1 election: Requested vote from peers cab11df9b0724ca7ba16a8938196064c (127.0.97.3:37435), 4c5946ff59fc46d7a89d744559e88e9a (127.0.97.2:46517)
I20260928 08:43:55.638729 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "236b5aff0aef48249fc312c9a8b3d1ce" candidate_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab11df9b0724ca7ba16a8938196064c"
I20260928 08:43:55.638742 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "236b5aff0aef48249fc312c9a8b3d1ce" candidate_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5946ff59fc46d7a89d744559e88e9a"
I20260928 08:43:55.638936 1811 raft_consensus.cc:3037] T 236b5aff0aef48249fc312c9a8b3d1ce P 4c5946ff59fc46d7a89d744559e88e9a [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:43:55.638933 1954 raft_consensus.cc:3037] T 236b5aff0aef48249fc312c9a8b3d1ce P cab11df9b0724ca7ba16a8938196064c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:43:55.641214 1811 raft_consensus.cc:2454] T 236b5aff0aef48249fc312c9a8b3d1ce P 4c5946ff59fc46d7a89d744559e88e9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c97af2a0a5a448c99b40aaf2e1bd520 in term 1.
I20260928 08:43:55.641309 1954 raft_consensus.cc:2454] T 236b5aff0aef48249fc312c9a8b3d1ce P cab11df9b0724ca7ba16a8938196064c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c97af2a0a5a448c99b40aaf2e1bd520 in term 1.
I20260928 08:43:55.641659 1601 leader_election.cc:302] T 236b5aff0aef48249fc312c9a8b3d1ce P 2c97af2a0a5a448c99b40aaf2e1bd520 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c97af2a0a5a448c99b40aaf2e1bd520, 4c5946ff59fc46d7a89d744559e88e9a; no voters:
I20260928 08:43:55.642032 2024 raft_consensus.cc:2781] T 236b5aff0aef48249fc312c9a8b3d1ce P 2c97af2a0a5a448c99b40aaf2e1bd520 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:43:55.642477 2024 raft_consensus.cc:686] T 236b5aff0aef48249fc312c9a8b3d1ce P 2c97af2a0a5a448c99b40aaf2e1bd520 [term 1 LEADER]: Becoming Leader. State: Replica: 2c97af2a0a5a448c99b40aaf2e1bd520, State: Running, Role: LEADER
I20260928 08:43:55.642999 2024 consensus_queue.cc:237] T 236b5aff0aef48249fc312c9a8b3d1ce P 2c97af2a0a5a448c99b40aaf2e1bd520 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } }
I20260928 08:43:55.648046 1513 catalog_manager.cc:5495] T 236b5aff0aef48249fc312c9a8b3d1ce P 2c97af2a0a5a448c99b40aaf2e1bd520 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c97af2a0a5a448c99b40aaf2e1bd520 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } health_report { overall_health: HEALTHY } } }
I20260928 08:43:55.696866 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:55.691547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47916 (local address 127.0.97.1:39109)
0928 08:43:55.691758 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:55.691762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:55.691866 (+ 104us) server_negotiation.cc:407] Connection header received
0928 08:43:55.692169 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:55.692173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:55.692269 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:55.692411 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:43:55.692831 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.693611 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.694851 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.695234 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.695332 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:55.695761 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:43:55.696359 (+ 598us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:43:55.696469 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:55.696597 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 08:43:55.696659 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 08:43:55.704368 1811 raft_consensus.cc:1261] T 236b5aff0aef48249fc312c9a8b3d1ce P 4c5946ff59fc46d7a89d744559e88e9a [term 1 FOLLOWER]: Refusing update from remote peer 2c97af2a0a5a448c99b40aaf2e1bd520: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:43:55.704422 1954 raft_consensus.cc:1261] T 236b5aff0aef48249fc312c9a8b3d1ce P cab11df9b0724ca7ba16a8938196064c [term 1 FOLLOWER]: Refusing update from remote peer 2c97af2a0a5a448c99b40aaf2e1bd520: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:43:55.705314 2029 consensus_queue.cc:1035] T 236b5aff0aef48249fc312c9a8b3d1ce P 2c97af2a0a5a448c99b40aaf2e1bd520 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:43:55.705626 2024 consensus_queue.cc:1035] T 236b5aff0aef48249fc312c9a8b3d1ce P 2c97af2a0a5a448c99b40aaf2e1bd520 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:43:55.716917 2032 mvcc.cc:205] Tried to move back new op lower bound from 7334236306236407808 to 7334236305997656064. Current Snapshot: MvccSnapshot[applied={T|T < 7334236306236407808}]
I20260928 08:43:55.720810 2034 mvcc.cc:205] Tried to move back new op lower bound from 7334236306236407808 to 7334236305997656064. Current Snapshot: MvccSnapshot[applied={T|T < 7334236306236407808}]
I20260928 08:43:55.739295 1514 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:43:55.739574 1514 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:43:55.743706 1514 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d5d3dbb83324641a6fc9c439b1bd217: Sending DeleteTablet for 3 replicas of tablet 236b5aff0aef48249fc312c9a8b3d1ce
I20260928 08:43:55.745011 1929 tablet_service.cc:1514] Processing DeleteTablet for tablet 236b5aff0aef48249fc312c9a8b3d1ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:43:55 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44362
I20260928 08:43:55.745163 1790 tablet_service.cc:1514] Processing DeleteTablet for tablet 236b5aff0aef48249fc312c9a8b3d1ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:43:55 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42078
I20260928 08:43:55.745272 1651 tablet_service.cc:1514] Processing DeleteTablet for tablet 236b5aff0aef48249fc312c9a8b3d1ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:43:55 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47908
I20260928 08:43:55.748602 1513 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:43:55.749249 1513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:43:55.758320 1789 tablet_service.cc:1467] Processing CreateTablet for tablet 8b92f2e776364148a9cf9a29b3ccf01a (DEFAULT_TABLE table=test-table [id=2b31a0b4eae746e991bdbf1540cd73a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:43:55.759341 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b92f2e776364148a9cf9a29b3ccf01a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:43:55.759441 2041 tablet_replica.cc:330] T 236b5aff0aef48249fc312c9a8b3d1ce P cab11df9b0724ca7ba16a8938196064c: stopping tablet replica
I20260928 08:43:55.760149 2041 raft_consensus.cc:2229] T 236b5aff0aef48249fc312c9a8b3d1ce P cab11df9b0724ca7ba16a8938196064c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:43:55.760607 2042 tablet_replica.cc:330] T 236b5aff0aef48249fc312c9a8b3d1ce P 4c5946ff59fc46d7a89d744559e88e9a: stopping tablet replica
I20260928 08:43:55.760653 2041 raft_consensus.cc:2258] T 236b5aff0aef48249fc312c9a8b3d1ce P cab11df9b0724ca7ba16a8938196064c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:43:55.761081 2042 raft_consensus.cc:2229] T 236b5aff0aef48249fc312c9a8b3d1ce P 4c5946ff59fc46d7a89d744559e88e9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:43:55.761446 2042 raft_consensus.cc:2258] T 236b5aff0aef48249fc312c9a8b3d1ce P 4c5946ff59fc46d7a89d744559e88e9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:43:55.763430 2042 ts_tablet_manager.cc:1905] T 236b5aff0aef48249fc312c9a8b3d1ce P 4c5946ff59fc46d7a89d744559e88e9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:43:55.763422 2041 ts_tablet_manager.cc:1905] T 236b5aff0aef48249fc312c9a8b3d1ce P cab11df9b0724ca7ba16a8938196064c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:43:55.765631 1650 tablet_service.cc:1467] Processing CreateTablet for tablet 8b92f2e776364148a9cf9a29b3ccf01a (DEFAULT_TABLE table=test-table [id=2b31a0b4eae746e991bdbf1540cd73a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:43:55.765722 1929 tablet_service.cc:1467] Processing CreateTablet for tablet 8b92f2e776364148a9cf9a29b3ccf01a (DEFAULT_TABLE table=test-table [id=2b31a0b4eae746e991bdbf1540cd73a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:43:55.766187 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b92f2e776364148a9cf9a29b3ccf01a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:43:55.766255 1650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b92f2e776364148a9cf9a29b3ccf01a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:43:55.770913 2041 ts_tablet_manager.cc:1918] T 236b5aff0aef48249fc312c9a8b3d1ce P cab11df9b0724ca7ba16a8938196064c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:43:55.771055 2041 log.cc:1196] T 236b5aff0aef48249fc312c9a8b3d1ce P cab11df9b0724ca7ba16a8938196064c: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-2/wal/wals/236b5aff0aef48249fc312c9a8b3d1ce
I20260928 08:43:55.771564 2041 ts_tablet_manager.cc:1939] T 236b5aff0aef48249fc312c9a8b3d1ce P cab11df9b0724ca7ba16a8938196064c: Deleting consensus metadata
I20260928 08:43:55.773640 2019 tablet_bootstrap.cc:492] T 8b92f2e776364148a9cf9a29b3ccf01a P 2c97af2a0a5a448c99b40aaf2e1bd520: Bootstrap starting.
I20260928 08:43:55.773789 1500 catalog_manager.cc:4841] TS cab11df9b0724ca7ba16a8938196064c (127.0.97.3:37435): tablet 236b5aff0aef48249fc312c9a8b3d1ce (table test-table [id=98dd05032e9b4cb69541649c5091aa30]) successfully deleted
I20260928 08:43:55.775475 2019 tablet_bootstrap.cc:654] T 8b92f2e776364148a9cf9a29b3ccf01a P 2c97af2a0a5a448c99b40aaf2e1bd520: Neither blocks nor log segments found. Creating new log.
I20260928 08:43:55.780932 2042 ts_tablet_manager.cc:1918] T 236b5aff0aef48249fc312c9a8b3d1ce P 4c5946ff59fc46d7a89d744559e88e9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:43:55.781176 2042 log.cc:1196] T 236b5aff0aef48249fc312c9a8b3d1ce P 4c5946ff59fc46d7a89d744559e88e9a: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-1/wal/wals/236b5aff0aef48249fc312c9a8b3d1ce
I20260928 08:43:55.781834 2042 ts_tablet_manager.cc:1939] T 236b5aff0aef48249fc312c9a8b3d1ce P 4c5946ff59fc46d7a89d744559e88e9a: Deleting consensus metadata
I20260928 08:43:55.783838 2020 tablet_bootstrap.cc:492] T 8b92f2e776364148a9cf9a29b3ccf01a P 4c5946ff59fc46d7a89d744559e88e9a: Bootstrap starting.
I20260928 08:43:55.784145 1498 catalog_manager.cc:4841] TS 4c5946ff59fc46d7a89d744559e88e9a (127.0.97.2:46517): tablet 236b5aff0aef48249fc312c9a8b3d1ce (table test-table [id=98dd05032e9b4cb69541649c5091aa30]) successfully deleted
I20260928 08:43:55.785560 2020 tablet_bootstrap.cc:654] T 8b92f2e776364148a9cf9a29b3ccf01a P 4c5946ff59fc46d7a89d744559e88e9a: Neither blocks nor log segments found. Creating new log.
I20260928 08:43:55.786118 2019 tablet_bootstrap.cc:492] T 8b92f2e776364148a9cf9a29b3ccf01a P 2c97af2a0a5a448c99b40aaf2e1bd520: No bootstrap required, opened a new log
I20260928 08:43:55.786262 2019 ts_tablet_manager.cc:1397] T 8b92f2e776364148a9cf9a29b3ccf01a P 2c97af2a0a5a448c99b40aaf2e1bd520: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260928 08:43:55.787114 2018 tablet_bootstrap.cc:492] T 8b92f2e776364148a9cf9a29b3ccf01a P cab11df9b0724ca7ba16a8938196064c: Bootstrap starting.
I20260928 08:43:55.789683 2018 tablet_bootstrap.cc:654] T 8b92f2e776364148a9cf9a29b3ccf01a P cab11df9b0724ca7ba16a8938196064c: Neither blocks nor log segments found. Creating new log.
I20260928 08:43:55.791286 2019 raft_consensus.cc:348] T 8b92f2e776364148a9cf9a29b3ccf01a P 2c97af2a0a5a448c99b40aaf2e1bd520 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } }
I20260928 08:43:55.792387 2019 raft_consensus.cc:374] T 8b92f2e776364148a9cf9a29b3ccf01a P 2c97af2a0a5a448c99b40aaf2e1bd520 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:43:55.792718 2019 raft_consensus.cc:729] T 8b92f2e776364148a9cf9a29b3ccf01a P 2c97af2a0a5a448c99b40aaf2e1bd520 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c97af2a0a5a448c99b40aaf2e1bd520, State: Initialized, Role: FOLLOWER
I20260928 08:43:55.793439 2019 consensus_queue.cc:260] T 8b92f2e776364148a9cf9a29b3ccf01a P 2c97af2a0a5a448c99b40aaf2e1bd520 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } }
I20260928 08:43:55.795431 2019 ts_tablet_manager.cc:1428] T 8b92f2e776364148a9cf9a29b3ccf01a P 2c97af2a0a5a448c99b40aaf2e1bd520: Time spent starting tablet: real 0.009s user 0.001s sys 0.002s
I20260928 08:43:55.791781 2043 tablet_replica.cc:330] T 236b5aff0aef48249fc312c9a8b3d1ce P 2c97af2a0a5a448c99b40aaf2e1bd520: stopping tablet replica
I20260928 08:43:55.796581 2043 raft_consensus.cc:2229] T 236b5aff0aef48249fc312c9a8b3d1ce P 2c97af2a0a5a448c99b40aaf2e1bd520 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:43:55.796901 2018 tablet_bootstrap.cc:492] T 8b92f2e776364148a9cf9a29b3ccf01a P cab11df9b0724ca7ba16a8938196064c: No bootstrap required, opened a new log
I20260928 08:43:55.797048 2018 ts_tablet_manager.cc:1397] T 8b92f2e776364148a9cf9a29b3ccf01a P cab11df9b0724ca7ba16a8938196064c: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260928 08:43:55.797402 2043 raft_consensus.cc:2258] T 236b5aff0aef48249fc312c9a8b3d1ce P 2c97af2a0a5a448c99b40aaf2e1bd520 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:43:55.798110 2018 raft_consensus.cc:348] T 8b92f2e776364148a9cf9a29b3ccf01a P cab11df9b0724ca7ba16a8938196064c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } }
I20260928 08:43:55.798382 2018 raft_consensus.cc:374] T 8b92f2e776364148a9cf9a29b3ccf01a P cab11df9b0724ca7ba16a8938196064c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:43:55.798398 2020 tablet_bootstrap.cc:492] T 8b92f2e776364148a9cf9a29b3ccf01a P 4c5946ff59fc46d7a89d744559e88e9a: No bootstrap required, opened a new log
I20260928 08:43:55.798535 2018 raft_consensus.cc:729] T 8b92f2e776364148a9cf9a29b3ccf01a P cab11df9b0724ca7ba16a8938196064c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab11df9b0724ca7ba16a8938196064c, State: Initialized, Role: FOLLOWER
I20260928 08:43:55.798561 2020 ts_tablet_manager.cc:1397] T 8b92f2e776364148a9cf9a29b3ccf01a P 4c5946ff59fc46d7a89d744559e88e9a: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.009s
I20260928 08:43:55.798904 2018 consensus_queue.cc:260] T 8b92f2e776364148a9cf9a29b3ccf01a P cab11df9b0724ca7ba16a8938196064c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } }
I20260928 08:43:55.800273 2020 raft_consensus.cc:348] T 8b92f2e776364148a9cf9a29b3ccf01a P 4c5946ff59fc46d7a89d744559e88e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } }
I20260928 08:43:55.800530 2020 raft_consensus.cc:374] T 8b92f2e776364148a9cf9a29b3ccf01a P 4c5946ff59fc46d7a89d744559e88e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:43:55.800619 2020 raft_consensus.cc:729] T 8b92f2e776364148a9cf9a29b3ccf01a P 4c5946ff59fc46d7a89d744559e88e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c5946ff59fc46d7a89d744559e88e9a, State: Initialized, Role: FOLLOWER
I20260928 08:43:55.800881 2020 consensus_queue.cc:260] T 8b92f2e776364148a9cf9a29b3ccf01a P 4c5946ff59fc46d7a89d744559e88e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } }
I20260928 08:43:55.801815 2018 ts_tablet_manager.cc:1428] T 8b92f2e776364148a9cf9a29b3ccf01a P cab11df9b0724ca7ba16a8938196064c: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20260928 08:43:55.802922 2043 ts_tablet_manager.cc:1905] T 236b5aff0aef48249fc312c9a8b3d1ce P 2c97af2a0a5a448c99b40aaf2e1bd520: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:43:55.803936 2020 ts_tablet_manager.cc:1428] T 8b92f2e776364148a9cf9a29b3ccf01a P 4c5946ff59fc46d7a89d744559e88e9a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260928 08:43:55.809266 2043 ts_tablet_manager.cc:1918] T 236b5aff0aef48249fc312c9a8b3d1ce P 2c97af2a0a5a448c99b40aaf2e1bd520: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:43:55.809438 2043 log.cc:1196] T 236b5aff0aef48249fc312c9a8b3d1ce P 2c97af2a0a5a448c99b40aaf2e1bd520: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-0/wal/wals/236b5aff0aef48249fc312c9a8b3d1ce
I20260928 08:43:55.809965 2043 ts_tablet_manager.cc:1939] T 236b5aff0aef48249fc312c9a8b3d1ce P 2c97af2a0a5a448c99b40aaf2e1bd520: Deleting consensus metadata
I20260928 08:43:55.811487 1499 catalog_manager.cc:4841] TS 2c97af2a0a5a448c99b40aaf2e1bd520 (127.0.97.1:39109): tablet 236b5aff0aef48249fc312c9a8b3d1ce (table test-table [id=98dd05032e9b4cb69541649c5091aa30]) successfully deleted
I20260928 08:43:55.936975 2025 raft_consensus.cc:482] T 8b92f2e776364148a9cf9a29b3ccf01a P 4c5946ff59fc46d7a89d744559e88e9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:43:55.937216 2025 raft_consensus.cc:504] T 8b92f2e776364148a9cf9a29b3ccf01a P 4c5946ff59fc46d7a89d744559e88e9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } }
I20260928 08:43:55.938791 2025 leader_election.cc:288] T 8b92f2e776364148a9cf9a29b3ccf01a P 4c5946ff59fc46d7a89d744559e88e9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c97af2a0a5a448c99b40aaf2e1bd520 (127.0.97.1:39109), cab11df9b0724ca7ba16a8938196064c (127.0.97.3:37435)
I20260928 08:43:55.948655 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:55.939242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51173 (local address 127.0.97.3:37435)
0928 08:43:55.939491 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:55.939500 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:55.940485 (+ 985us) server_negotiation.cc:407] Connection header received
0928 08:43:55.940636 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:55.940641 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:55.940723 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:55.940804 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:43:55.941599 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.942726 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.946443 (+ 3717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.947419 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.947524 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:55.947632 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:55.948227 (+ 595us) server_negotiation.cc:299] Negotiation successful
0928 08:43:55.948373 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 08:43:55.948742 2047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:55.939055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39109 (local address 127.0.97.2:44217)
0928 08:43:55.940031 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:55.940054 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:55.940259 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:55.940922 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:55.940925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:55.940944 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:43:55.941361 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:55.941369 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.942885 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.942891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:55.946206 (+ 3315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:55.946220 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.947977 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.947982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:55.948147 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:55.948186 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 08:43:55.948267 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 08:43:55.948386 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"mutex_wait_us":240,"thread_start_us":660,"threads_started":1}
I20260928 08:43:55.948755 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:55.939515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44217 (local address 127.0.97.1:39109)
0928 08:43:55.939702 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:55.939708 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:55.940144 (+ 436us) server_negotiation.cc:407] Connection header received
0928 08:43:55.940410 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:55.940414 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:55.940501 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:55.940608 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:43:55.941526 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.942726 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.946400 (+ 3674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.947691 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.947861 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:55.947955 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:55.948314 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 08:43:55.948448 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 08:43:55.949677 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:55.939079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37435 (local address 127.0.97.2:51173)
0928 08:43:55.940372 (+ 1293us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:55.940389 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:55.940513 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:55.940859 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:55.940865 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:55.940895 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:43:55.941436 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:55.941450 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.942885 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.942891 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:55.946206 (+ 3315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:55.946220 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:55.947807 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:55.947814 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:55.948035 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:55.948074 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 08:43:55.949239 (+ 1165us) client_negotiation.cc:240] Negotiation successful
0928 08:43:55.949358 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1159,"mutex_wait_us":311,"thread_start_us":949,"threads_started":1}
I20260928 08:43:55.949889 1675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b92f2e776364148a9cf9a29b3ccf01a" candidate_uuid: "4c5946ff59fc46d7a89d744559e88e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" is_pre_election: true
I20260928 08:43:55.950402 1675 raft_consensus.cc:2454] T 8b92f2e776364148a9cf9a29b3ccf01a P 2c97af2a0a5a448c99b40aaf2e1bd520 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c5946ff59fc46d7a89d744559e88e9a in term 0.
I20260928 08:43:55.950438 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b92f2e776364148a9cf9a29b3ccf01a" candidate_uuid: "4c5946ff59fc46d7a89d744559e88e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab11df9b0724ca7ba16a8938196064c" is_pre_election: true
I20260928 08:43:55.950693 1954 raft_consensus.cc:2454] T 8b92f2e776364148a9cf9a29b3ccf01a P cab11df9b0724ca7ba16a8938196064c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c5946ff59fc46d7a89d744559e88e9a in term 0.
I20260928 08:43:55.951077 1742 leader_election.cc:302] T 8b92f2e776364148a9cf9a29b3ccf01a P 4c5946ff59fc46d7a89d744559e88e9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c97af2a0a5a448c99b40aaf2e1bd520, 4c5946ff59fc46d7a89d744559e88e9a; no voters:
I20260928 08:43:55.951500 2025 raft_consensus.cc:2781] T 8b92f2e776364148a9cf9a29b3ccf01a P 4c5946ff59fc46d7a89d744559e88e9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:43:55.951622 2025 raft_consensus.cc:482] T 8b92f2e776364148a9cf9a29b3ccf01a P 4c5946ff59fc46d7a89d744559e88e9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:43:55.951762 2025 raft_consensus.cc:3037] T 8b92f2e776364148a9cf9a29b3ccf01a P 4c5946ff59fc46d7a89d744559e88e9a [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:43:55.954607 2025 raft_consensus.cc:504] T 8b92f2e776364148a9cf9a29b3ccf01a P 4c5946ff59fc46d7a89d744559e88e9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } }
I20260928 08:43:55.955257 2025 leader_election.cc:288] T 8b92f2e776364148a9cf9a29b3ccf01a P 4c5946ff59fc46d7a89d744559e88e9a [CANDIDATE]: Term 1 election: Requested vote from peers 2c97af2a0a5a448c99b40aaf2e1bd520 (127.0.97.1:39109), cab11df9b0724ca7ba16a8938196064c (127.0.97.3:37435)
I20260928 08:43:55.955693 1675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b92f2e776364148a9cf9a29b3ccf01a" candidate_uuid: "4c5946ff59fc46d7a89d744559e88e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520"
I20260928 08:43:55.955929 1675 raft_consensus.cc:3037] T 8b92f2e776364148a9cf9a29b3ccf01a P 2c97af2a0a5a448c99b40aaf2e1bd520 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:43:55.955865 1954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b92f2e776364148a9cf9a29b3ccf01a" candidate_uuid: "4c5946ff59fc46d7a89d744559e88e9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cab11df9b0724ca7ba16a8938196064c"
I20260928 08:43:55.956087 1954 raft_consensus.cc:3037] T 8b92f2e776364148a9cf9a29b3ccf01a P cab11df9b0724ca7ba16a8938196064c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:43:55.958250 1675 raft_consensus.cc:2454] T 8b92f2e776364148a9cf9a29b3ccf01a P 2c97af2a0a5a448c99b40aaf2e1bd520 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c5946ff59fc46d7a89d744559e88e9a in term 1.
I20260928 08:43:55.958624 1742 leader_election.cc:302] T 8b92f2e776364148a9cf9a29b3ccf01a P 4c5946ff59fc46d7a89d744559e88e9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c97af2a0a5a448c99b40aaf2e1bd520, 4c5946ff59fc46d7a89d744559e88e9a; no voters:
I20260928 08:43:55.958895 2025 raft_consensus.cc:2781] T 8b92f2e776364148a9cf9a29b3ccf01a P 4c5946ff59fc46d7a89d744559e88e9a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:43:55.959028 1954 raft_consensus.cc:2454] T 8b92f2e776364148a9cf9a29b3ccf01a P cab11df9b0724ca7ba16a8938196064c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c5946ff59fc46d7a89d744559e88e9a in term 1.
I20260928 08:43:55.959486 2025 raft_consensus.cc:686] T 8b92f2e776364148a9cf9a29b3ccf01a P 4c5946ff59fc46d7a89d744559e88e9a [term 1 LEADER]: Becoming Leader. State: Replica: 4c5946ff59fc46d7a89d744559e88e9a, State: Running, Role: LEADER
I20260928 08:43:55.960196 2025 consensus_queue.cc:237] T 8b92f2e776364148a9cf9a29b3ccf01a P 4c5946ff59fc46d7a89d744559e88e9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } }
I20260928 08:43:55.963940 1513 catalog_manager.cc:5495] T 8b92f2e776364148a9cf9a29b3ccf01a P 4c5946ff59fc46d7a89d744559e88e9a reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c5946ff59fc46d7a89d744559e88e9a (127.0.97.2). New cstate: current_term: 1 leader_uuid: "4c5946ff59fc46d7a89d744559e88e9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } health_report { overall_health: UNKNOWN } } }
Sep 28 08:43:56 dist-test-slave-n6xs krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:43:56.053579 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:56.038256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35969 (local address 127.0.0.1:49692)
0928 08:43:56.039428 (+ 1172us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:56.039455 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:56.039624 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:56.040006 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:56.040013 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:56.040744 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:43:56.040955 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:56.040965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.041963 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.041970 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:56.042920 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:56.042931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.043631 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.043637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:56.043796 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:56.044769 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:43:56.044786 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:43:56.047117 (+ 2331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:43:56.050546 (+ 3429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:56.050558 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:56.050566 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:56.051151 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:56.051543 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:56.051549 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:56.051553 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:56.051737 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:56.052403 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:43:56.052411 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:43:56.052577 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 08:43:56.052746 (+ 169us) client_negotiation.cc:240] Negotiation successful
0928 08:43:56.053065 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":968,"thread_start_us":357,"threads_started":1}
I20260928 08:43:56.053622 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:56.038312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49692 (local address 127.0.97.62:35969)
0928 08:43:56.039653 (+ 1341us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:56.039660 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:56.039686 (+ 26us) server_negotiation.cc:407] Connection header received
0928 08:43:56.039737 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:56.039743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:56.039799 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:56.039968 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:56.041097 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.041843 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.043068 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.043454 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.043611 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:56.047355 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:56.047360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:56.047363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:56.047391 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:56.050392 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:56.051259 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:56.051265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:56.051269 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:56.051441 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:56.051861 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:56.051865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:56.051869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:56.052268 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:56.052425 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:56.052880 (+ 455us) server_negotiation.cc:299] Negotiation successful
0928 08:43:56.053232 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":122,"server-negotiator.queue_time_us":1232,"thread_start_us":1084,"threads_started":1}
I20260928 08:43:56.060479 1513 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49692:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 08:43:56.061563 1513 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:43:56.068840 1789 tablet_service.cc:1467] Processing CreateTablet for tablet e0fee72634494b14b67c898e5710d979 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=febdfa60a89143fe86f1e6acd4612dbf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:43:56.069006 1651 tablet_service.cc:1467] Processing CreateTablet for tablet e0fee72634494b14b67c898e5710d979 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=febdfa60a89143fe86f1e6acd4612dbf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:43:56.069392 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0fee72634494b14b67c898e5710d979. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:43:56.069595 1651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0fee72634494b14b67c898e5710d979. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:43:56.074396 2020 tablet_bootstrap.cc:492] T e0fee72634494b14b67c898e5710d979 P 4c5946ff59fc46d7a89d744559e88e9a: Bootstrap starting.
I20260928 08:43:56.075703 2020 tablet_bootstrap.cc:654] T e0fee72634494b14b67c898e5710d979 P 4c5946ff59fc46d7a89d744559e88e9a: Neither blocks nor log segments found. Creating new log.
I20260928 08:43:56.077893 2019 tablet_bootstrap.cc:492] T e0fee72634494b14b67c898e5710d979 P 2c97af2a0a5a448c99b40aaf2e1bd520: Bootstrap starting.
I20260928 08:43:56.077955 1929 tablet_service.cc:1467] Processing CreateTablet for tablet e0fee72634494b14b67c898e5710d979 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=febdfa60a89143fe86f1e6acd4612dbf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:43:56.078416 1929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0fee72634494b14b67c898e5710d979. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:43:56.079756 2019 tablet_bootstrap.cc:654] T e0fee72634494b14b67c898e5710d979 P 2c97af2a0a5a448c99b40aaf2e1bd520: Neither blocks nor log segments found. Creating new log.
I20260928 08:43:56.082589 2019 tablet_bootstrap.cc:492] T e0fee72634494b14b67c898e5710d979 P 2c97af2a0a5a448c99b40aaf2e1bd520: No bootstrap required, opened a new log
I20260928 08:43:56.083194 2019 ts_tablet_manager.cc:1397] T e0fee72634494b14b67c898e5710d979 P 2c97af2a0a5a448c99b40aaf2e1bd520: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 08:43:56.084069 2019 raft_consensus.cc:348] T e0fee72634494b14b67c898e5710d979 P 2c97af2a0a5a448c99b40aaf2e1bd520 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } }
I20260928 08:43:56.084409 2019 raft_consensus.cc:374] T e0fee72634494b14b67c898e5710d979 P 2c97af2a0a5a448c99b40aaf2e1bd520 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:43:56.084524 2019 raft_consensus.cc:729] T e0fee72634494b14b67c898e5710d979 P 2c97af2a0a5a448c99b40aaf2e1bd520 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c97af2a0a5a448c99b40aaf2e1bd520, State: Initialized, Role: FOLLOWER
I20260928 08:43:56.084580 2020 tablet_bootstrap.cc:492] T e0fee72634494b14b67c898e5710d979 P 4c5946ff59fc46d7a89d744559e88e9a: No bootstrap required, opened a new log
I20260928 08:43:56.084735 2020 ts_tablet_manager.cc:1397] T e0fee72634494b14b67c898e5710d979 P 4c5946ff59fc46d7a89d744559e88e9a: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20260928 08:43:56.084829 2019 consensus_queue.cc:260] T e0fee72634494b14b67c898e5710d979 P 2c97af2a0a5a448c99b40aaf2e1bd520 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } }
I20260928 08:43:56.085507 2019 ts_tablet_manager.cc:1428] T e0fee72634494b14b67c898e5710d979 P 2c97af2a0a5a448c99b40aaf2e1bd520: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:43:56.085345 2024 tablet_replica.cc:439] T e0fee72634494b14b67c898e5710d979 P 2c97af2a0a5a448c99b40aaf2e1bd520: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } } }
I20260928 08:43:56.085690 2024 tablet_replica.cc:442] T e0fee72634494b14b67c898e5710d979 P 2c97af2a0a5a448c99b40aaf2e1bd520: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:43:56.086220 2020 raft_consensus.cc:348] T e0fee72634494b14b67c898e5710d979 P 4c5946ff59fc46d7a89d744559e88e9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } }
I20260928 08:43:56.086437 2020 raft_consensus.cc:374] T e0fee72634494b14b67c898e5710d979 P 4c5946ff59fc46d7a89d744559e88e9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:43:56.086519 2020 raft_consensus.cc:729] T e0fee72634494b14b67c898e5710d979 P 4c5946ff59fc46d7a89d744559e88e9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c5946ff59fc46d7a89d744559e88e9a, State: Initialized, Role: FOLLOWER
I20260928 08:43:56.086783 2020 consensus_queue.cc:260] T e0fee72634494b14b67c898e5710d979 P 4c5946ff59fc46d7a89d744559e88e9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } }
I20260928 08:43:56.087172 2025 tablet_replica.cc:439] T e0fee72634494b14b67c898e5710d979 P 4c5946ff59fc46d7a89d744559e88e9a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } } }
I20260928 08:43:56.087368 2020 ts_tablet_manager.cc:1428] T e0fee72634494b14b67c898e5710d979 P 4c5946ff59fc46d7a89d744559e88e9a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 08:43:56.087464 2025 tablet_replica.cc:442] T e0fee72634494b14b67c898e5710d979 P 4c5946ff59fc46d7a89d744559e88e9a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:43:56.090416 2018 tablet_bootstrap.cc:492] T e0fee72634494b14b67c898e5710d979 P cab11df9b0724ca7ba16a8938196064c: Bootstrap starting.
I20260928 08:43:56.092453 2018 tablet_bootstrap.cc:654] T e0fee72634494b14b67c898e5710d979 P cab11df9b0724ca7ba16a8938196064c: Neither blocks nor log segments found. Creating new log.
I20260928 08:43:56.094496 2018 tablet_bootstrap.cc:492] T e0fee72634494b14b67c898e5710d979 P cab11df9b0724ca7ba16a8938196064c: No bootstrap required, opened a new log
I20260928 08:43:56.094657 2018 ts_tablet_manager.cc:1397] T e0fee72634494b14b67c898e5710d979 P cab11df9b0724ca7ba16a8938196064c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 08:43:56.095270 2018 raft_consensus.cc:348] T e0fee72634494b14b67c898e5710d979 P cab11df9b0724ca7ba16a8938196064c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } }
I20260928 08:43:56.095445 2018 raft_consensus.cc:374] T e0fee72634494b14b67c898e5710d979 P cab11df9b0724ca7ba16a8938196064c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:43:56.095539 2018 raft_consensus.cc:729] T e0fee72634494b14b67c898e5710d979 P cab11df9b0724ca7ba16a8938196064c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab11df9b0724ca7ba16a8938196064c, State: Initialized, Role: FOLLOWER
I20260928 08:43:56.095769 2018 consensus_queue.cc:260] T e0fee72634494b14b67c898e5710d979 P cab11df9b0724ca7ba16a8938196064c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } }
I20260928 08:43:56.096346 2018 ts_tablet_manager.cc:1428] T e0fee72634494b14b67c898e5710d979 P cab11df9b0724ca7ba16a8938196064c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 08:43:56.096246 2026 tablet_replica.cc:439] T e0fee72634494b14b67c898e5710d979 P cab11df9b0724ca7ba16a8938196064c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } } }
I20260928 08:43:56.096627 2026 tablet_replica.cc:442] T e0fee72634494b14b67c898e5710d979 P cab11df9b0724ca7ba16a8938196064c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:43:56.236577 2026 raft_consensus.cc:482] T e0fee72634494b14b67c898e5710d979 P cab11df9b0724ca7ba16a8938196064c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:43:56.236784 2026 raft_consensus.cc:504] T e0fee72634494b14b67c898e5710d979 P cab11df9b0724ca7ba16a8938196064c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } }
I20260928 08:43:56.238682 2026 leader_election.cc:288] T e0fee72634494b14b67c898e5710d979 P cab11df9b0724ca7ba16a8938196064c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c97af2a0a5a448c99b40aaf2e1bd520 (127.0.97.1:39109), 4c5946ff59fc46d7a89d744559e88e9a (127.0.97.2:46517)
I20260928 08:43:56.248230 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:56.240192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35215 (local address 127.0.97.1:39109)
0928 08:43:56.240707 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:56.240713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:56.241229 (+ 516us) server_negotiation.cc:407] Connection header received
0928 08:43:56.241494 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:56.241500 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:56.241565 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:56.241618 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:43:56.242320 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.243413 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.245937 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.247137 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.247226 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:56.247322 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:56.247835 (+ 513us) server_negotiation.cc:299] Negotiation successful
0928 08:43:56.247980 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383}
I20260928 08:43:56.249239 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:56.238971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39109 (local address 127.0.97.3:35215)
0928 08:43:56.241139 (+ 2168us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:56.241158 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:56.241331 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:56.241781 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:56.241786 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:56.241818 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:43:56.242139 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:56.242148 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.243578 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.243583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:56.245742 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:56.245755 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.247441 (+ 1686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.247447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:56.247635 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:56.247668 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 08:43:56.248649 (+ 981us) client_negotiation.cc:240] Negotiation successful
0928 08:43:56.248788 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2035,"thread_start_us":1942,"threads_started":1}
I20260928 08:43:56.250375 1675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0fee72634494b14b67c898e5710d979" candidate_uuid: "cab11df9b0724ca7ba16a8938196064c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" is_pre_election: true
I20260928 08:43:56.250692 1675 raft_consensus.cc:2454] T e0fee72634494b14b67c898e5710d979 P 2c97af2a0a5a448c99b40aaf2e1bd520 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab11df9b0724ca7ba16a8938196064c in term 0.
I20260928 08:43:56.251518 1881 leader_election.cc:302] T e0fee72634494b14b67c898e5710d979 P cab11df9b0724ca7ba16a8938196064c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c97af2a0a5a448c99b40aaf2e1bd520, cab11df9b0724ca7ba16a8938196064c; no voters:
I20260928 08:43:56.251974 2026 raft_consensus.cc:2781] T e0fee72634494b14b67c898e5710d979 P cab11df9b0724ca7ba16a8938196064c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:43:56.252131 2026 raft_consensus.cc:482] T e0fee72634494b14b67c898e5710d979 P cab11df9b0724ca7ba16a8938196064c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:43:56.252305 2026 raft_consensus.cc:3037] T e0fee72634494b14b67c898e5710d979 P cab11df9b0724ca7ba16a8938196064c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:43:56.256011 2026 raft_consensus.cc:504] T e0fee72634494b14b67c898e5710d979 P cab11df9b0724ca7ba16a8938196064c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } }
I20260928 08:43:56.256803 2026 leader_election.cc:288] T e0fee72634494b14b67c898e5710d979 P cab11df9b0724ca7ba16a8938196064c [CANDIDATE]: Term 1 election: Requested vote from peers 2c97af2a0a5a448c99b40aaf2e1bd520 (127.0.97.1:39109), 4c5946ff59fc46d7a89d744559e88e9a (127.0.97.2:46517)
I20260928 08:43:56.257210 1675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0fee72634494b14b67c898e5710d979" candidate_uuid: "cab11df9b0724ca7ba16a8938196064c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520"
I20260928 08:43:56.257452 1675 raft_consensus.cc:3037] T e0fee72634494b14b67c898e5710d979 P 2c97af2a0a5a448c99b40aaf2e1bd520 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:43:56.257756 2061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:56.239122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46517 (local address 127.0.97.3:51997)
0928 08:43:56.249620 (+ 10498us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:56.249638 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:56.249736 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:56.251270 (+ 1534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:56.251276 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:56.251300 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:43:56.251468 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:56.251476 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.253002 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.253015 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:56.255206 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:56.255221 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.257035 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.257041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:56.257219 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:56.257250 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 08:43:56.257338 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 08:43:56.257478 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10401,"thread_start_us":11878,"threads_started":1}
I20260928 08:43:56.258064 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:56.239557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51997 (local address 127.0.97.2:46517)
0928 08:43:56.250631 (+ 11074us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:56.250637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:56.250657 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:43:56.250732 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:56.250736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:56.250810 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:56.251614 (+ 804us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:43:56.251758 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.252840 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.255422 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.256705 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.256832 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:56.256950 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:56.257417 (+ 467us) server_negotiation.cc:299] Negotiation successful
0928 08:43:56.257584 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10937,"thread_start_us":10799,"threads_started":1}
I20260928 08:43:56.258839 1811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0fee72634494b14b67c898e5710d979" candidate_uuid: "cab11df9b0724ca7ba16a8938196064c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5946ff59fc46d7a89d744559e88e9a" is_pre_election: true
I20260928 08:43:56.258960 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0fee72634494b14b67c898e5710d979" candidate_uuid: "cab11df9b0724ca7ba16a8938196064c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c5946ff59fc46d7a89d744559e88e9a"
I20260928 08:43:56.259087 1811 raft_consensus.cc:2454] T e0fee72634494b14b67c898e5710d979 P 4c5946ff59fc46d7a89d744559e88e9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cab11df9b0724ca7ba16a8938196064c in term 0.
I20260928 08:43:56.260650 1675 raft_consensus.cc:2454] T e0fee72634494b14b67c898e5710d979 P 2c97af2a0a5a448c99b40aaf2e1bd520 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cab11df9b0724ca7ba16a8938196064c in term 1.
I20260928 08:43:56.261171 1881 leader_election.cc:302] T e0fee72634494b14b67c898e5710d979 P cab11df9b0724ca7ba16a8938196064c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2c97af2a0a5a448c99b40aaf2e1bd520, cab11df9b0724ca7ba16a8938196064c; no voters: 4c5946ff59fc46d7a89d744559e88e9a
I20260928 08:43:56.261514 2026 raft_consensus.cc:2781] T e0fee72634494b14b67c898e5710d979 P cab11df9b0724ca7ba16a8938196064c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:43:56.262184 2026 raft_consensus.cc:686] T e0fee72634494b14b67c898e5710d979 P cab11df9b0724ca7ba16a8938196064c [term 1 LEADER]: Becoming Leader. State: Replica: cab11df9b0724ca7ba16a8938196064c, State: Running, Role: LEADER
I20260928 08:43:56.262609 2026 consensus_queue.cc:237] T e0fee72634494b14b67c898e5710d979 P cab11df9b0724ca7ba16a8938196064c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } }
I20260928 08:43:56.265633 2064 tablet_replica.cc:439] T e0fee72634494b14b67c898e5710d979 P cab11df9b0724ca7ba16a8938196064c: TxnStatusTablet state changed. Reason: New leader cab11df9b0724ca7ba16a8938196064c. Latest consensus state: current_term: 1 leader_uuid: "cab11df9b0724ca7ba16a8938196064c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } } }
I20260928 08:43:56.266086 2064 tablet_replica.cc:442] T e0fee72634494b14b67c898e5710d979 P cab11df9b0724ca7ba16a8938196064c: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:43:56.266721 2066 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 08:43:56.266932 2066 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 08:43:56.267263 1514 catalog_manager.cc:5495] T e0fee72634494b14b67c898e5710d979 P cab11df9b0724ca7ba16a8938196064c reported cstate change: term changed from 0 to 1, leader changed from <none> to cab11df9b0724ca7ba16a8938196064c (127.0.97.3). New cstate: current_term: 1 leader_uuid: "cab11df9b0724ca7ba16a8938196064c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } health_report { overall_health: HEALTHY } } }
I20260928 08:43:56.291215 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:56.284929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37435 (local address 127.0.0.1:44378)
0928 08:43:56.285167 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:56.285184 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:56.285298 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:56.285778 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:56.285781 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:56.285790 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:43:56.285976 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:56.285984 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.287486 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.287490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:56.288562 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:56.288573 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.289517 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.289521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:56.289656 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:56.289706 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:43:56.290749 (+ 1043us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:43:56.290764 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 08:43:56.290869 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 08:43:56.290979 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 08:43:56.291285 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:56.285040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44378 (local address 127.0.97.3:37435)
0928 08:43:56.285223 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:56.285229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:56.285250 (+ 21us) server_negotiation.cc:407] Connection header received
0928 08:43:56.285497 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:56.285502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:56.285607 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:56.285762 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:43:56.286148 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.287338 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.288697 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.289363 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.289499 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:56.289889 (+ 390us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:43:56.290625 (+ 736us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:43:56.290801 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:56.290915 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 08:43:56.291000 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 08:43:56.296272 1809 raft_consensus.cc:3037] T e0fee72634494b14b67c898e5710d979 P 4c5946ff59fc46d7a89d744559e88e9a [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:43:56.296576 1675 raft_consensus.cc:1261] T e0fee72634494b14b67c898e5710d979 P 2c97af2a0a5a448c99b40aaf2e1bd520 [term 1 FOLLOWER]: Refusing update from remote peer cab11df9b0724ca7ba16a8938196064c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:43:56.297415 2064 consensus_queue.cc:1035] T e0fee72634494b14b67c898e5710d979 P cab11df9b0724ca7ba16a8938196064c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:43:56.299693 1809 raft_consensus.cc:1261] T e0fee72634494b14b67c898e5710d979 P 4c5946ff59fc46d7a89d744559e88e9a [term 1 FOLLOWER]: Refusing update from remote peer cab11df9b0724ca7ba16a8938196064c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:43:56.300736 2064 consensus_queue.cc:1035] T e0fee72634494b14b67c898e5710d979 P cab11df9b0724ca7ba16a8938196064c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:43:56.301201 2024 tablet_replica.cc:439] T e0fee72634494b14b67c898e5710d979 P 2c97af2a0a5a448c99b40aaf2e1bd520: TxnStatusTablet state changed. Reason: New leader cab11df9b0724ca7ba16a8938196064c. Latest consensus state: current_term: 1 leader_uuid: "cab11df9b0724ca7ba16a8938196064c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } } }
I20260928 08:43:56.301505 2024 tablet_replica.cc:442] T e0fee72634494b14b67c898e5710d979 P 2c97af2a0a5a448c99b40aaf2e1bd520: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:43:56.303663 2069 mvcc.cc:205] Tried to move back new op lower bound from 7334236308662325248 to 7334236308535517184. Current Snapshot: MvccSnapshot[applied={T|T < 7334236308662325248}]
I20260928 08:43:56.308640 2026 tablet_replica.cc:439] T e0fee72634494b14b67c898e5710d979 P cab11df9b0724ca7ba16a8938196064c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cab11df9b0724ca7ba16a8938196064c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } } }
I20260928 08:43:56.308971 2026 tablet_replica.cc:442] T e0fee72634494b14b67c898e5710d979 P cab11df9b0724ca7ba16a8938196064c: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:43:56.314612 2024 tablet_replica.cc:439] T e0fee72634494b14b67c898e5710d979 P 2c97af2a0a5a448c99b40aaf2e1bd520: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cab11df9b0724ca7ba16a8938196064c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } } }
I20260928 08:43:56.314859 2024 tablet_replica.cc:442] T e0fee72634494b14b67c898e5710d979 P 2c97af2a0a5a448c99b40aaf2e1bd520: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:43:56.321530 2025 tablet_replica.cc:439] T e0fee72634494b14b67c898e5710d979 P 4c5946ff59fc46d7a89d744559e88e9a: TxnStatusTablet state changed. Reason: New leader cab11df9b0724ca7ba16a8938196064c. Latest consensus state: current_term: 1 leader_uuid: "cab11df9b0724ca7ba16a8938196064c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } } }
I20260928 08:43:56.321866 2025 tablet_replica.cc:442] T e0fee72634494b14b67c898e5710d979 P 4c5946ff59fc46d7a89d744559e88e9a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:43:56.324044 2064 tablet_replica.cc:439] T e0fee72634494b14b67c898e5710d979 P cab11df9b0724ca7ba16a8938196064c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cab11df9b0724ca7ba16a8938196064c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } } }
I20260928 08:43:56.324329 2064 tablet_replica.cc:442] T e0fee72634494b14b67c898e5710d979 P cab11df9b0724ca7ba16a8938196064c: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:43:56.327086 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:56.318189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42094 (local address 127.0.97.2:46517)
0928 08:43:56.318372 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:56.318376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:56.318940 (+ 564us) server_negotiation.cc:407] Connection header received
0928 08:43:56.319230 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:56.319234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:56.319349 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:56.319506 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:43:56.320171 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.321085 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.324378 (+ 3293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.324863 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.325002 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:56.325492 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:43:56.326392 (+ 900us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:43:56.326541 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:56.326725 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 08:43:56.326803 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":52}
I20260928 08:43:56.336992 2025 tablet_replica.cc:439] T e0fee72634494b14b67c898e5710d979 P 4c5946ff59fc46d7a89d744559e88e9a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cab11df9b0724ca7ba16a8938196064c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 } } peers { permanent_uuid: "4c5946ff59fc46d7a89d744559e88e9a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46517 } } peers { permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 } } }
I20260928 08:43:56.337330 2025 tablet_replica.cc:442] T e0fee72634494b14b67c898e5710d979 P 4c5946ff59fc46d7a89d744559e88e9a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:43:56.343901 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:56.337773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49063 (local address 127.0.97.3:37435)
0928 08:43:56.337940 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:56.337946 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:56.338105 (+ 159us) server_negotiation.cc:407] Connection header received
0928 08:43:56.338375 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:56.338389 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:56.338473 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:56.338570 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:43:56.339171 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.340315 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.341711 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.342494 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.342625 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:56.343053 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:43:56.343229 (+ 176us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:43:56.343325 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:56.343507 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 08:43:56.343584 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 08:43:56.343935 2081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:56.337423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37435 (local address 127.0.97.2:49063)
0928 08:43:56.338006 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:56.338026 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:56.338214 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:56.338620 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:56.338626 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:56.338641 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:43:56.339015 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:56.339027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.340471 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.340475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:56.341561 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:56.341572 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.342684 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.342689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:56.342820 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:56.342873 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:43:56.343359 (+ 486us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:43:56.343371 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 08:43:56.343452 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 08:43:56.343563 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":532,"threads_started":1}
I20260928 08:43:56.353008 1954 raft_consensus.cc:1261] T 8b92f2e776364148a9cf9a29b3ccf01a P cab11df9b0724ca7ba16a8938196064c [term 1 FOLLOWER]: Refusing update from remote peer 4c5946ff59fc46d7a89d744559e88e9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:43:56.353190 1675 raft_consensus.cc:1261] T 8b92f2e776364148a9cf9a29b3ccf01a P 2c97af2a0a5a448c99b40aaf2e1bd520 [term 1 FOLLOWER]: Refusing update from remote peer 4c5946ff59fc46d7a89d744559e88e9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:43:56.353762 2025 consensus_queue.cc:1035] T 8b92f2e776364148a9cf9a29b3ccf01a P 4c5946ff59fc46d7a89d744559e88e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cab11df9b0724ca7ba16a8938196064c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:43:56.354082 2049 consensus_queue.cc:1035] T 8b92f2e776364148a9cf9a29b3ccf01a P 4c5946ff59fc46d7a89d744559e88e9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c97af2a0a5a448c99b40aaf2e1bd520" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:43:56.396807 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:56.389183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56371 (local address 127.0.97.2:46517)
0928 08:43:56.389368 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:56.389374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:56.390273 (+ 899us) server_negotiation.cc:407] Connection header received
0928 08:43:56.390817 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:56.390823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:56.390911 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:56.391008 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:43:56.391593 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.392498 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.394967 (+ 2469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.395444 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.395596 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:56.396092 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:43:56.396202 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:43:56.396298 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:56.396407 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 08:43:56.396480 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 08:43:56.396929 2086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:56.389103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46517 (local address 127.0.97.3:56371)
0928 08:43:56.390186 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:56.390208 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:56.390354 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:56.391025 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:56.391029 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:56.391041 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:43:56.391429 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:56.391441 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.392619 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.392626 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:56.394833 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:56.394845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.395627 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.395634 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:56.395786 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:56.395906 (+ 120us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:43:56.396297 (+ 391us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:43:56.396310 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 08:43:56.396388 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 08:43:56.396534 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"thread_start_us":655,"threads_started":1}
I20260928 08:43:56.432854 1514 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:43:56.433120 1514 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:43:56.436555 1514 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d5d3dbb83324641a6fc9c439b1bd217: Sending DeleteTablet for 3 replicas of tablet 8b92f2e776364148a9cf9a29b3ccf01a
I20260928 08:43:56.437619 1651 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b92f2e776364148a9cf9a29b3ccf01a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:43:56 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47908
I20260928 08:43:56.438139 1789 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b92f2e776364148a9cf9a29b3ccf01a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:43:56 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42078
I20260928 08:43:56.438175 1929 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b92f2e776364148a9cf9a29b3ccf01a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:43:56 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44362
I20260928 08:43:56.450707 2093 tablet_replica.cc:330] T 8b92f2e776364148a9cf9a29b3ccf01a P 4c5946ff59fc46d7a89d744559e88e9a: stopping tablet replica
I20260928 08:43:56.451063 2093 raft_consensus.cc:2229] T 8b92f2e776364148a9cf9a29b3ccf01a P 4c5946ff59fc46d7a89d744559e88e9a [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:43:56.451741 2093 raft_consensus.cc:2258] T 8b92f2e776364148a9cf9a29b3ccf01a P 4c5946ff59fc46d7a89d744559e88e9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:43:56.453068 2093 ts_tablet_manager.cc:1905] T 8b92f2e776364148a9cf9a29b3ccf01a P 4c5946ff59fc46d7a89d744559e88e9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:43:56.460965 2093 ts_tablet_manager.cc:1918] T 8b92f2e776364148a9cf9a29b3ccf01a P 4c5946ff59fc46d7a89d744559e88e9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:43:56.461138 2093 log.cc:1196] T 8b92f2e776364148a9cf9a29b3ccf01a P 4c5946ff59fc46d7a89d744559e88e9a: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-1/wal/wals/8b92f2e776364148a9cf9a29b3ccf01a
I20260928 08:43:56.461620 2093 ts_tablet_manager.cc:1939] T 8b92f2e776364148a9cf9a29b3ccf01a P 4c5946ff59fc46d7a89d744559e88e9a: Deleting consensus metadata
I20260928 08:43:56.462821 1498 catalog_manager.cc:4841] TS 4c5946ff59fc46d7a89d744559e88e9a (127.0.97.2:46517): tablet 8b92f2e776364148a9cf9a29b3ccf01a (table test-table [id=2b31a0b4eae746e991bdbf1540cd73a6]) successfully deleted
I20260928 08:43:56.466373 2091 tablet_replica.cc:330] T 8b92f2e776364148a9cf9a29b3ccf01a P 2c97af2a0a5a448c99b40aaf2e1bd520: stopping tablet replica
I20260928 08:43:56.466952 2092 tablet_replica.cc:330] T 8b92f2e776364148a9cf9a29b3ccf01a P cab11df9b0724ca7ba16a8938196064c: stopping tablet replica
I20260928 08:43:56.467281 2091 raft_consensus.cc:2229] T 8b92f2e776364148a9cf9a29b3ccf01a P 2c97af2a0a5a448c99b40aaf2e1bd520 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:43:56.467551 2092 raft_consensus.cc:2229] T 8b92f2e776364148a9cf9a29b3ccf01a P cab11df9b0724ca7ba16a8938196064c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:43:56.467875 2091 raft_consensus.cc:2258] T 8b92f2e776364148a9cf9a29b3ccf01a P 2c97af2a0a5a448c99b40aaf2e1bd520 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:43:56.468173 2092 raft_consensus.cc:2258] T 8b92f2e776364148a9cf9a29b3ccf01a P cab11df9b0724ca7ba16a8938196064c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:43:56.469550 2091 ts_tablet_manager.cc:1905] T 8b92f2e776364148a9cf9a29b3ccf01a P 2c97af2a0a5a448c99b40aaf2e1bd520: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:43:56.474509 2092 ts_tablet_manager.cc:1905] T 8b92f2e776364148a9cf9a29b3ccf01a P cab11df9b0724ca7ba16a8938196064c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:43:56.477064 2091 ts_tablet_manager.cc:1918] T 8b92f2e776364148a9cf9a29b3ccf01a P 2c97af2a0a5a448c99b40aaf2e1bd520: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:43:56.477244 2091 log.cc:1196] T 8b92f2e776364148a9cf9a29b3ccf01a P 2c97af2a0a5a448c99b40aaf2e1bd520: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-0/wal/wals/8b92f2e776364148a9cf9a29b3ccf01a
I20260928 08:43:56.477725 2091 ts_tablet_manager.cc:1939] T 8b92f2e776364148a9cf9a29b3ccf01a P 2c97af2a0a5a448c99b40aaf2e1bd520: Deleting consensus metadata
I20260928 08:43:56.478992 1499 catalog_manager.cc:4841] TS 2c97af2a0a5a448c99b40aaf2e1bd520 (127.0.97.1:39109): tablet 8b92f2e776364148a9cf9a29b3ccf01a (table test-table [id=2b31a0b4eae746e991bdbf1540cd73a6]) successfully deleted
Sep 28 08:43:56 dist-test-slave-n6xs krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585035, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:43:56.482465 2092 ts_tablet_manager.cc:1918] T 8b92f2e776364148a9cf9a29b3ccf01a P cab11df9b0724ca7ba16a8938196064c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:43:56.482692 2092 log.cc:1196] T 8b92f2e776364148a9cf9a29b3ccf01a P cab11df9b0724ca7ba16a8938196064c: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790585022442112-388-0/minicluster-data/ts-2/wal/wals/8b92f2e776364148a9cf9a29b3ccf01a
I20260928 08:43:56.483196 2092 ts_tablet_manager.cc:1939] T 8b92f2e776364148a9cf9a29b3ccf01a P cab11df9b0724ca7ba16a8938196064c: Deleting consensus metadata
I20260928 08:43:56.484856 1500 catalog_manager.cc:4841] TS cab11df9b0724ca7ba16a8938196064c (127.0.97.3:37435): tablet 8b92f2e776364148a9cf9a29b3ccf01a (table test-table [id=2b31a0b4eae746e991bdbf1540cd73a6]) successfully deleted
I20260928 08:43:56.489225 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:56.448898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47932 (local address 127.0.97.1:39109)
0928 08:43:56.449165 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:56.449174 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:56.465417 (+ 16243us) server_negotiation.cc:407] Connection header received
0928 08:43:56.465946 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:56.465952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:56.466050 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:56.466126 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:56.468812 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.469949 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.472306 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.472977 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.473078 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:56.481767 (+ 8689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:56.481783 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:56.481803 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:56.481845 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:56.485796 (+ 3951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:56.486736 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:56.486747 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:56.486751 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:56.486907 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:56.487347 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:56.487353 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:56.487356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:56.487834 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:56.488156 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:56.488546 (+ 390us) server_negotiation.cc:299] Negotiation successful
0928 08:43:56.488807 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260928 08:43:56.490097 1716 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47932
I20260928 08:43:56.518193 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:56.501161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49702 (local address 127.0.97.62:35969)
0928 08:43:56.501344 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:56.501351 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:56.502051 (+ 700us) server_negotiation.cc:407] Connection header received
0928 08:43:56.502239 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:56.502244 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:56.502363 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:56.502482 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:56.504161 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.505416 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.507212 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.508033 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.508173 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:56.511154 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:56.511161 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:56.511167 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:56.511211 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:56.515051 (+ 3840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:56.515890 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:56.515897 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:56.515901 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:56.516051 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:56.516477 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:56.516483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:56.516486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:56.516908 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:56.517177 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:56.517565 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 08:43:56.517816 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 08:43:56.518993 1514 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49702
I20260928 08:43:56.521637 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 1585
I20260928 08:43:56.540274 2017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:56.524381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47942 (local address 127.0.97.1:39109)
0928 08:43:56.524562 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:56.524570 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:56.524588 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:43:56.524658 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:56.524663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:56.524742 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:56.524858 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:56.526606 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.527877 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.529674 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.530341 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.530473 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:56.533330 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:56.533337 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:56.533343 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:56.533384 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:56.537202 (+ 3818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:56.538007 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:56.538014 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:56.538018 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:56.538183 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:56.538615 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:56.538621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:56.538625 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:56.538987 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:56.539272 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:56.539578 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 08:43:56.539843 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 08:43:56.541172 1716 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47942
W20260928 08:43:56.541836 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 1585: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:43:56.542065 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 1585
I20260928 08:43:56.574986 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 1725
Sep 28 08:43:56 dist-test-slave-n6xs krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585035, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 08:43:56.595957 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:56.576234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42096 (local address 127.0.97.2:46517)
0928 08:43:56.576420 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:56.576427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:56.576634 (+ 207us) server_negotiation.cc:407] Connection header received
0928 08:43:56.576789 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:56.576795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:56.576901 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:56.577017 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:56.578611 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.579801 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.581748 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.582465 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.582603 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:56.588105 (+ 5502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:56.588122 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:56.588141 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:56.588187 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:56.592282 (+ 4095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:56.593268 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:56.593277 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:56.593280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:56.593472 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:56.593935 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:56.593942 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:56.593946 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:56.594470 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:56.594815 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:56.595104 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 08:43:56.595367 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260928 08:43:56.596679 1855 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42096
W20260928 08:43:56.597469 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 1725: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:43:56.597618 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 1725
I20260928 08:43:56.618644 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 1864
Sep 28 08:43:56 dist-test-slave-n6xs krb5kdc[1469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585035, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 08:43:56.636101 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:56.619389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44394 (local address 127.0.97.3:37435)
0928 08:43:56.619584 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:56.619589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:56.620094 (+ 505us) server_negotiation.cc:407] Connection header received
0928 08:43:56.620435 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:56.620442 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:56.620587 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:56.620743 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:56.622125 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.622955 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.624326 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:56.624787 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:56.624940 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:56.629366 (+ 4426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:56.629378 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:56.629393 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:56.629432 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:56.633044 (+ 3612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:56.633869 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:56.633877 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:56.633880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:56.634034 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:56.634467 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:56.634471 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:56.634473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:56.634833 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:56.635197 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:56.635411 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 08:43:56.635633 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 08:43:56.636899 1994 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44394
W20260928 08:43:56.637629 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 1864: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:43:56.637831 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 1864
I20260928 08:43:56.657646 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 1482
I20260928 08:43:56.658684 1554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:43:56.769362 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 1482
2026-09-28T08:43:56Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4882 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:43:56 dist-test-slave-n6xs krb5kdc[2102](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:43:56 dist-test-slave-n6xs krb5kdc[2102](info): set up 2 sockets
Sep 28 08:43:56 dist-test-slave-n6xs krb5kdc[2102](info): commencing operation
krb5kdc: starting...
W20260928 08:43:57.863826 388 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:43:57 dist-test-slave-n6xs krb5kdc[2102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:43:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:43:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:43:58.053503 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42255
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:43:58.365417 2115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:43:58.365708 2115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:43:58.365878 2115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:43:58.373147 2115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:43:58.373256 2115 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 08:43:58.373325 2115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:43:58.373416 2115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:43:58.373495 2115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:43:58.373610 2115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:43:58.383478 2115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42255
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:43:58.384776 2115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:43:58.386132 2115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:43:58.398058 2120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:58.398096 2121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:58.398135 2123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:43:58.399003 2115 server_base.cc:1034] running on GCE node
I20260928 08:43:58.399768 2115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:43:58.401152 2115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:43:58.402377 2115 hybrid_clock.cc:648] HybridClock initialized: now 1790585038402324 us; error 34 us; skew 500 ppm
I20260928 08:43:58.402725 2115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:43:58 dist-test-slave-n6xs krb5kdc[2102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:43:58.405874 2115 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:43:58.407238 2115 webserver.cc:460] Webserver started at http://127.0.97.62:34297/ using document root <none> and password file <none>
I20260928 08:43:58.407989 2115 fs_manager.cc:359] Metadata directory not provided
I20260928 08:43:58.408099 2115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:43:58.408421 2115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:43:58.411507 2115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "353e51706cce4c289e9293253cc0d1dd"
format_stamp: "Formatted at 2026-09-28 08:43:58 on dist-test-slave-n6xs"
server_key: "4b6052c497b87c1dc98a72dd8476dfdd"
server_key_iv: "cf1373f67c93d29d1d99bda35ba89b85"
server_key_version: "encryptionkey@0"
I20260928 08:43:58.412358 2115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "353e51706cce4c289e9293253cc0d1dd"
format_stamp: "Formatted at 2026-09-28 08:43:58 on dist-test-slave-n6xs"
server_key: "4b6052c497b87c1dc98a72dd8476dfdd"
server_key_iv: "cf1373f67c93d29d1d99bda35ba89b85"
server_key_version: "encryptionkey@0"
I20260928 08:43:58.417266 2115 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 08:43:58.420531 2130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:43:58.421839 2115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:43:58.422020 2115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "353e51706cce4c289e9293253cc0d1dd"
format_stamp: "Formatted at 2026-09-28 08:43:58 on dist-test-slave-n6xs"
server_key: "4b6052c497b87c1dc98a72dd8476dfdd"
server_key_iv: "cf1373f67c93d29d1d99bda35ba89b85"
server_key_version: "encryptionkey@0"
I20260928 08:43:58.422230 2115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:43:58.443562 2115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:43:58.447603 2115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:43:58.447906 2115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:43:58.469482 2115 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42255
I20260928 08:43:58.469501 2191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42255 every 8 connection(s)
I20260928 08:43:58.470927 2115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:43:58.474687 2192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:43:58.480926 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 2115
I20260928 08:43:58.481179 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:43:58.481864 388 external_mini_cluster.cc:1428] Setting key 614a78eebd925637e3a058f7ae5cf5f7
I20260928 08:43:58.485155 2192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 353e51706cce4c289e9293253cc0d1dd: Bootstrap starting.
I20260928 08:43:58.488705 2192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 353e51706cce4c289e9293253cc0d1dd: Neither blocks nor log segments found. Creating new log.
I20260928 08:43:58.489842 2192 log.cc:824] T 00000000000000000000000000000000 P 353e51706cce4c289e9293253cc0d1dd: Log is configured to *not* fsync() on all Append() calls
I20260928 08:43:58.493095 2192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 353e51706cce4c289e9293253cc0d1dd: No bootstrap required, opened a new log
Sep 28 08:43:58 dist-test-slave-n6xs krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:43:58.503592 2192 raft_consensus.cc:348] T 00000000000000000000000000000000 P 353e51706cce4c289e9293253cc0d1dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353e51706cce4c289e9293253cc0d1dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42255 } }
I20260928 08:43:58.504072 2192 raft_consensus.cc:374] T 00000000000000000000000000000000 P 353e51706cce4c289e9293253cc0d1dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:43:58.504179 2192 raft_consensus.cc:729] T 00000000000000000000000000000000 P 353e51706cce4c289e9293253cc0d1dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 353e51706cce4c289e9293253cc0d1dd, State: Initialized, Role: FOLLOWER
I20260928 08:43:58.504858 2192 consensus_queue.cc:260] T 00000000000000000000000000000000 P 353e51706cce4c289e9293253cc0d1dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353e51706cce4c289e9293253cc0d1dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42255 } }
I20260928 08:43:58.505157 2192 raft_consensus.cc:388] T 00000000000000000000000000000000 P 353e51706cce4c289e9293253cc0d1dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:43:58.505308 2192 raft_consensus.cc:482] T 00000000000000000000000000000000 P 353e51706cce4c289e9293253cc0d1dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:43:58.505484 2192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 353e51706cce4c289e9293253cc0d1dd [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:43:58.507069 2195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:58.483983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36964 (local address 127.0.97.62:42255)
0928 08:43:58.484765 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:58.484777 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:58.484821 (+ 44us) server_negotiation.cc:407] Connection header received
0928 08:43:58.486557 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:58.486584 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:58.486986 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:58.487341 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:58.489262 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:58.491012 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:58.492762 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:58.493520 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:58.493664 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:58.498350 (+ 4686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:58.498362 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:58.498383 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:58.498436 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:58.502460 (+ 4024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:58.503149 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:58.503159 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:58.503169 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:58.503370 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:58.503746 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:58.503751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:58.503753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:58.504451 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:58.504756 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:58.505207 (+ 451us) server_negotiation.cc:299] Negotiation successful
0928 08:43:58.505670 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":377,"threads_started":1}
I20260928 08:43:58.508715 2192 raft_consensus.cc:504] T 00000000000000000000000000000000 P 353e51706cce4c289e9293253cc0d1dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353e51706cce4c289e9293253cc0d1dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42255 } }
I20260928 08:43:58.509213 2192 leader_election.cc:302] T 00000000000000000000000000000000 P 353e51706cce4c289e9293253cc0d1dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 353e51706cce4c289e9293253cc0d1dd; no voters:
I20260928 08:43:58.509858 2192 leader_election.cc:288] T 00000000000000000000000000000000 P 353e51706cce4c289e9293253cc0d1dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:43:58.510032 2197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 353e51706cce4c289e9293253cc0d1dd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:43:58.510703 2197 raft_consensus.cc:686] T 00000000000000000000000000000000 P 353e51706cce4c289e9293253cc0d1dd [term 1 LEADER]: Becoming Leader. State: Replica: 353e51706cce4c289e9293253cc0d1dd, State: Running, Role: LEADER
I20260928 08:43:58.511232 2197 consensus_queue.cc:237] T 00000000000000000000000000000000 P 353e51706cce4c289e9293253cc0d1dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353e51706cce4c289e9293253cc0d1dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42255 } }
I20260928 08:43:58.511924 2192 sys_catalog.cc:564] T 00000000000000000000000000000000 P 353e51706cce4c289e9293253cc0d1dd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:43:58.514406 2199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 353e51706cce4c289e9293253cc0d1dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 353e51706cce4c289e9293253cc0d1dd. Latest consensus state: current_term: 1 leader_uuid: "353e51706cce4c289e9293253cc0d1dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353e51706cce4c289e9293253cc0d1dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42255 } } }
I20260928 08:43:58.514392 2198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 353e51706cce4c289e9293253cc0d1dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "353e51706cce4c289e9293253cc0d1dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353e51706cce4c289e9293253cc0d1dd" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42255 } } }
I20260928 08:43:58.514834 2199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 353e51706cce4c289e9293253cc0d1dd [sys.catalog]: This master's current role is: LEADER
I20260928 08:43:58.514834 2198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 353e51706cce4c289e9293253cc0d1dd [sys.catalog]: This master's current role is: LEADER
I20260928 08:43:58.516974 2204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:43:58.522135 2204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:43:58.531121 2204 catalog_manager.cc:1347] Generated new cluster ID: bcd9304873134e3198d79c892539856e
I20260928 08:43:58.531236 2204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:43:58.548547 2204 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:43:58.549850 2204 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:43:58.562547 2204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 353e51706cce4c289e9293253cc0d1dd: Generated new TSK 0
I20260928 08:43:58.563632 2204 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:43:58.621203 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42255
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:43:58.939066 2218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:43:58.939338 2218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:43:58.939508 2218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:43:58.946681 2218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:43:58.946820 2218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:43:58.946978 2218 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 08:43:58.947057 2218 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:43:58.947184 2218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:43:58.957129 2218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:43:58.958451 2218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:43:58.959759 2218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:43:58.972869 2223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:58.972915 2224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:58.972939 2226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:43:58.974009 2218 server_base.cc:1034] running on GCE node
I20260928 08:43:58.974525 2218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:43:58.975407 2218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:43:58.976596 2218 hybrid_clock.cc:648] HybridClock initialized: now 1790585038976545 us; error 39 us; skew 500 ppm
I20260928 08:43:58.976936 2218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:43:58 dist-test-slave-n6xs krb5kdc[2102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:43:58.980007 2218 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:43:58.981457 2218 webserver.cc:460] Webserver started at http://127.0.97.1:46295/ using document root <none> and password file <none>
I20260928 08:43:58.982122 2218 fs_manager.cc:359] Metadata directory not provided
I20260928 08:43:58.982240 2218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:43:58.982553 2218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:43:58.985471 2218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-0/data/instance:
uuid: "219436dd00ba4093adc396062d6ef10f"
format_stamp: "Formatted at 2026-09-28 08:43:58 on dist-test-slave-n6xs"
server_key: "0476ec109c9af5588b4291332c883ac6"
server_key_iv: "5f11eaa98da79a908ab5dbe69d49100f"
server_key_version: "encryptionkey@0"
I20260928 08:43:58.986274 2218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-0/wal/instance:
uuid: "219436dd00ba4093adc396062d6ef10f"
format_stamp: "Formatted at 2026-09-28 08:43:58 on dist-test-slave-n6xs"
server_key: "0476ec109c9af5588b4291332c883ac6"
server_key_iv: "5f11eaa98da79a908ab5dbe69d49100f"
server_key_version: "encryptionkey@0"
I20260928 08:43:58.991279 2218 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:43:58.994612 2233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:43:58.995930 2218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 08:43:58.996089 2218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-0/wal
uuid: "219436dd00ba4093adc396062d6ef10f"
format_stamp: "Formatted at 2026-09-28 08:43:58 on dist-test-slave-n6xs"
server_key: "0476ec109c9af5588b4291332c883ac6"
server_key_iv: "5f11eaa98da79a908ab5dbe69d49100f"
server_key_version: "encryptionkey@0"
I20260928 08:43:58.996279 2218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:43:59.028949 2218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:43:59.033069 2218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:43:59.033363 2218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:43:59.036023 2218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:43:59.036170 2218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:43:59.036299 2218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:43:59.036398 2218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:43:59 dist-test-slave-n6xs krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:43:59.060273 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:59.036899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37527 (local address 127.0.97.62:42255)
0928 08:43:59.037460 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:59.037466 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:59.038717 (+ 1251us) server_negotiation.cc:407] Connection header received
0928 08:43:59.041831 (+ 3114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:59.041838 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:59.041948 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:59.042082 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:59.044910 (+ 2828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:59.045905 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:59.048198 (+ 2293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:59.048610 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:59.048749 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:59.052736 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:59.052742 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:59.052746 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:59.052806 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:59.056116 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:59.057316 (+ 1200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:59.057322 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:59.057326 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:59.057454 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:59.058069 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:59.058073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:59.058077 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:59.058378 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:59.058488 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:59.059555 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0928 08:43:59.059766 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":328,"threads_started":1}
I20260928 08:43:59.062130 2243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:59.037555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42255 (local address 127.0.97.1:37527)
0928 08:43:59.038506 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:59.038560 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:59.041405 (+ 2845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:59.042399 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:59.042417 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:59.043501 (+ 1084us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:43:59.044682 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:59.044720 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:59.046115 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:59.046121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:59.048006 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:59.048027 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:59.048851 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:59.048860 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:59.049075 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:59.049849 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:43:59.049870 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:43:59.052485 (+ 2615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:43:59.056282 (+ 3797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:59.056297 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:59.056318 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:59.057166 (+ 848us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:59.057571 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:59.057579 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:59.057585 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:59.057913 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:59.058511 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:43:59.058523 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:43:59.059100 (+ 577us) client_negotiation.cc:769] Sending connection context
0928 08:43:59.059483 (+ 383us) client_negotiation.cc:240] Negotiation successful
0928 08:43:59.060243 (+ 760us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":428,"threads_started":1}
I20260928 08:43:59.083995 2242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:59.069163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:44809 (local address 127.0.97.62:42255)
0928 08:43:59.069388 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:59.069397 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:59.069918 (+ 521us) server_negotiation.cc:407] Connection header received
0928 08:43:59.070135 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:59.070143 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:59.070245 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:59.070374 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:59.071161 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:59.072488 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:59.075244 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:59.075794 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:59.075994 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:59.078610 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:59.078615 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:59.078618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:59.078644 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:59.081490 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:59.082113 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:59.082120 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:59.082123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:59.082254 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:59.082599 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:59.082606 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:59.082609 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:59.082854 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:59.083007 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:59.083390 (+ 383us) server_negotiation.cc:299] Negotiation successful
0928 08:43:59.083601 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 08:43:59.084136 2302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:59.069158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42255 (local address 127.0.97.1:44809)
0928 08:43:59.069807 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:59.069827 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:59.069995 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:59.070386 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:59.070389 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:59.070774 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:43:59.071028 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:59.071036 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:59.072678 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:59.072687 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:59.075057 (+ 2370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:59.075084 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:59.076059 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:59.076069 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:59.076341 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:59.077074 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:43:59.077105 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:43:59.078427 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:43:59.081617 (+ 3190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:59.081622 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:59.081626 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:59.082000 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:59.082368 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:59.082373 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:59.082376 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:59.082497 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:59.082967 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:43:59.082972 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:43:59.083164 (+ 192us) client_negotiation.cc:769] Sending connection context
0928 08:43:59.083308 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 08:43:59.083601 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":374,"threads_started":1}
I20260928 08:43:59.098042 2218 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45001
I20260928 08:43:59.098086 2353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45001 every 8 connection(s)
I20260928 08:43:59.099340 2218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:43:59.101673 2354 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42255
I20260928 08:43:59.101930 2354 heartbeater.cc:464] Registering TS with master...
I20260928 08:43:59.102582 2354 heartbeater.cc:511] Master 127.0.97.62:42255 requested a full tablet report, sending...
I20260928 08:43:59.104460 2147 ts_manager.cc:194] Registered new tserver with Master: 219436dd00ba4093adc396062d6ef10f (127.0.97.1:45001)
I20260928 08:43:59.105891 2147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:37527
I20260928 08:43:59.109239 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 2218
I20260928 08:43:59.109445 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:43:59.110026 388 external_mini_cluster.cc:1428] Setting key 2e5cc63ab6b0df72a168bb1906a210ec
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 08:43:59.150836 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42255
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:43:59.476121 2358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:43:59.476516 2358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:43:59.476668 2358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:43:59.488437 2358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:43:59.488588 2358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:43:59.488713 2358 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 08:43:59.488786 2358 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:43:59.488889 2358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:43:59.500362 2358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:43:59.501619 2358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:43:59.502880 2358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:43:59.516865 2364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:59.516863 2366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:43:59.516863 2363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:43:59.518363 2358 server_base.cc:1034] running on GCE node
I20260928 08:43:59.519023 2358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:43:59.520074 2358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:43:59.521281 2358 hybrid_clock.cc:648] HybridClock initialized: now 1790585039521231 us; error 41 us; skew 500 ppm
I20260928 08:43:59.521728 2358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:43:59 dist-test-slave-n6xs krb5kdc[2102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:43:59.525102 2358 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:43:59.526696 2358 webserver.cc:460] Webserver started at http://127.0.97.2:40235/ using document root <none> and password file <none>
I20260928 08:43:59.527484 2358 fs_manager.cc:359] Metadata directory not provided
I20260928 08:43:59.527590 2358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:43:59.527928 2358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:43:59.530856 2358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-1/data/instance:
uuid: "2a944f0f60ae43498dd4ef372b97458c"
format_stamp: "Formatted at 2026-09-28 08:43:59 on dist-test-slave-n6xs"
server_key: "9f2f65b748dde5a6ef283ec4e574c2da"
server_key_iv: "74680dfa6771a3f5d316b5147d9f6dd6"
server_key_version: "encryptionkey@0"
I20260928 08:43:59.531613 2358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-1/wal/instance:
uuid: "2a944f0f60ae43498dd4ef372b97458c"
format_stamp: "Formatted at 2026-09-28 08:43:59 on dist-test-slave-n6xs"
server_key: "9f2f65b748dde5a6ef283ec4e574c2da"
server_key_iv: "74680dfa6771a3f5d316b5147d9f6dd6"
server_key_version: "encryptionkey@0"
I20260928 08:43:59.537086 2358 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 08:43:59.540813 2373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:43:59.542240 2358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 08:43:59.542426 2358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-1/wal
uuid: "2a944f0f60ae43498dd4ef372b97458c"
format_stamp: "Formatted at 2026-09-28 08:43:59 on dist-test-slave-n6xs"
server_key: "9f2f65b748dde5a6ef283ec4e574c2da"
server_key_iv: "74680dfa6771a3f5d316b5147d9f6dd6"
server_key_version: "encryptionkey@0"
I20260928 08:43:59.542641 2358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:43:59.578533 2358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:43:59.582906 2358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:43:59.583199 2358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:43:59.586030 2358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:43:59.586165 2358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:43:59.586309 2358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:43:59.586407 2358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:43:59 dist-test-slave-n6xs krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585039, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:43:59.613663 2382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:59.586876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41871 (local address 127.0.97.62:42255)
0928 08:43:59.587314 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:59.587320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:59.588709 (+ 1389us) server_negotiation.cc:407] Connection header received
0928 08:43:59.592021 (+ 3312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:59.592030 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:59.592145 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:59.592289 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:59.594966 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:59.596144 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:59.598889 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:59.599713 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:59.600785 (+ 1072us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:59.605487 (+ 4702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:59.605497 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:59.605508 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:59.605562 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:59.609313 (+ 3751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:59.610449 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:59.610456 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:59.610460 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:59.610638 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:59.611291 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:59.611298 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:59.611301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:59.611683 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:59.611865 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:59.613063 (+ 1198us) server_negotiation.cc:299] Negotiation successful
0928 08:43:59.613316 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":214,"threads_started":1}
I20260928 08:43:59.615877 2383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:59.587558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42255 (local address 127.0.97.2:41871)
0928 08:43:59.588529 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:59.588580 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:59.591559 (+ 2979us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:59.592596 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:59.592612 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:59.593520 (+ 908us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:43:59.594685 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:59.594715 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:59.596376 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:59.596382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:59.598696 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:59.598714 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:59.600016 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:59.600026 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:59.600257 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:59.601157 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:43:59.601210 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:43:59.605169 (+ 3959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:43:59.609485 (+ 4316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:59.609498 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:59.609523 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:59.610305 (+ 782us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:59.610768 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:59.610773 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:59.610777 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:59.611151 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:59.611857 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:43:59.611870 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:43:59.612509 (+ 639us) client_negotiation.cc:769] Sending connection context
0928 08:43:59.612980 (+ 471us) client_negotiation.cc:240] Negotiation successful
0928 08:43:59.613738 (+ 758us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":430,"threads_started":1}
I20260928 08:43:59.638561 2382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:59.622251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42929 (local address 127.0.97.62:42255)
0928 08:43:59.622481 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 08:43:59.622490 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:43:59.623303 (+ 813us) server_negotiation.cc:407] Connection header received
0928 08:43:59.623463 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:43:59.623471 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:43:59.623591 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:43:59.623725 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:43:59.624789 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:59.626258 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:59.628000 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:43:59.628759 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:43:59.628909 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:59.631306 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:43:59.631314 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:43:59.631321 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:43:59.631364 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:43:59.634930 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:59.635997 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:59.636005 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:59.636010 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:59.636162 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:43:59.636772 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:43:59.636778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:43:59.636782 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:43:59.637251 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:43:59.637415 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 08:43:59.637924 (+ 509us) server_negotiation.cc:299] Negotiation successful
0928 08:43:59.638154 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 08:43:59.638818 2466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:43:59.622514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42255 (local address 127.0.97.2:42929)
0928 08:43:59.623184 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0928 08:43:59.623206 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:43:59.623343 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:43:59.623769 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:43:59.623772 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:43:59.624312 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:43:59.624627 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:59.624637 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:59.626507 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:59.626512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:59.627729 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:43:59.627742 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:43:59.628971 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:43:59.628976 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:43:59.629109 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:43:59.629861 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:43:59.629891 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:43:59.631049 (+ 1158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:43:59.635107 (+ 4058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:59.635114 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:59.635118 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:59.635858 (+ 740us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:59.636300 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:43:59.636310 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:43:59.636314 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:43:59.636568 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:43:59.637476 (+ 908us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:43:59.637481 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:43:59.637654 (+ 173us) client_negotiation.cc:769] Sending connection context
0928 08:43:59.637818 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 08:43:59.638214 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":369,"threads_started":1}
I20260928 08:43:59.640913 2358 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34607
I20260928 08:43:59.640950 2493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34607 every 8 connection(s)
I20260928 08:43:59.642223 2358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:43:59.644799 2494 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42255
I20260928 08:43:59.645047 2494 heartbeater.cc:464] Registering TS with master...
I20260928 08:43:59.645885 2494 heartbeater.cc:511] Master 127.0.97.62:42255 requested a full tablet report, sending...
I20260928 08:43:59.648836 2147 ts_manager.cc:194] Registered new tserver with Master: 2a944f0f60ae43498dd4ef372b97458c (127.0.97.2:34607)
I20260928 08:43:59.649402 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 2358
I20260928 08:43:59.649576 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:43:59.650008 2147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:41871
I20260928 08:43:59.650084 388 external_mini_cluster.cc:1428] Setting key b5054f9d62f7cf8cc50214eecf5ee8f0
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:43:59.691051 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42255
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:00.017632 2498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:00.017951 2498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:00.018136 2498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:00.025286 2498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:00.025450 2498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:00.025588 2498 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 08:44:00.025677 2498 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:44:00.025790 2498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:44:00.035560 2498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:42255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:00.036942 2498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:00.038285 2498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:00.051388 2506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:00.051337 2504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:00.051359 2503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:00.052275 2498 server_base.cc:1034] running on GCE node
I20260928 08:44:00.052851 2498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:00.053776 2498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:00.054991 2498 hybrid_clock.cc:648] HybridClock initialized: now 1790585040054925 us; error 53 us; skew 500 ppm
I20260928 08:44:00.055317 2498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:00 dist-test-slave-n6xs krb5kdc[2102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:00.062073 2498 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:44:00.063639 2498 webserver.cc:460] Webserver started at http://127.0.97.3:46115/ using document root <none> and password file <none>
I20260928 08:44:00.064502 2498 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:00.064617 2498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:00.064898 2498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:00.068111 2498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-2/data/instance:
uuid: "469696e9fb2948138ebce9b1bd275d5c"
format_stamp: "Formatted at 2026-09-28 08:44:00 on dist-test-slave-n6xs"
server_key: "8fa653e1952abba9ee116ac5ed265985"
server_key_iv: "396c8235a4fdb4a464e488735c7b8932"
server_key_version: "encryptionkey@0"
I20260928 08:44:00.068953 2498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-2/wal/instance:
uuid: "469696e9fb2948138ebce9b1bd275d5c"
format_stamp: "Formatted at 2026-09-28 08:44:00 on dist-test-slave-n6xs"
server_key: "8fa653e1952abba9ee116ac5ed265985"
server_key_iv: "396c8235a4fdb4a464e488735c7b8932"
server_key_version: "encryptionkey@0"
I20260928 08:44:00.074348 2498 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:44:00.078037 2513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:00.079457 2498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:44:00.079632 2498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-2/wal
uuid: "469696e9fb2948138ebce9b1bd275d5c"
format_stamp: "Formatted at 2026-09-28 08:44:00 on dist-test-slave-n6xs"
server_key: "8fa653e1952abba9ee116ac5ed265985"
server_key_iv: "396c8235a4fdb4a464e488735c7b8932"
server_key_version: "encryptionkey@0"
I20260928 08:44:00.079856 2498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:00.106827 2498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:00.108572 2354 heartbeater.cc:503] Master 127.0.97.62:42255 was elected leader, sending a full tablet report...
I20260928 08:44:00.111671 2498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:00.111989 2498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:00.114495 2498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:44:00.114624 2498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:00.114746 2498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:44:00.114835 2498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:44:00 dist-test-slave-n6xs krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585040, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:44:00.139833 2382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:00.115295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33047 (local address 127.0.97.62:42255)
0928 08:44:00.115489 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:00.115495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:00.117097 (+ 1602us) server_negotiation.cc:407] Connection header received
0928 08:44:00.119785 (+ 2688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:00.119790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:00.119902 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:00.120011 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:00.121727 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.122559 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.124559 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.125307 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.125474 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:00.131297 (+ 5823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:00.131306 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:00.131312 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:00.131366 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:00.135285 (+ 3919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:00.136400 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:00.136408 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:00.136423 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:00.136604 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:00.137303 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:00.137309 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:00.137315 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:00.137715 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:00.137879 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:00.138986 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0928 08:44:00.139283 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 08:44:00.142023 2522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:00.115910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42255 (local address 127.0.97.3:33047)
0928 08:44:00.116908 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:00.116959 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:00.119464 (+ 2505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:00.120315 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:00.120323 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:00.121031 (+ 708us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:00.121558 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:00.121574 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.122730 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.122738 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:00.124419 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:00.124430 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.125634 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.125647 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:00.126012 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:00.126878 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:00.126909 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:00.131006 (+ 4097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:00.135448 (+ 4442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:00.135460 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:00.135480 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:00.136252 (+ 772us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:00.136743 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:00.136750 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:00.136757 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:00.137144 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:00.137866 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:00.137879 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:00.138434 (+ 555us) client_negotiation.cc:769] Sending connection context
0928 08:44:00.138839 (+ 405us) client_negotiation.cc:240] Negotiation successful
0928 08:44:00.139681 (+ 842us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":459,"threads_started":1}
I20260928 08:44:00.159204 2498 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37393
I20260928 08:44:00.159286 2632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37393 every 8 connection(s)
I20260928 08:44:00.160691 2498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:44:00.162971 2633 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42255
I20260928 08:44:00.164152 2633 heartbeater.cc:464] Registering TS with master...
I20260928 08:44:00.165009 2633 heartbeater.cc:511] Master 127.0.97.62:42255 requested a full tablet report, sending...
I20260928 08:44:00.166182 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:00.149420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42255 (local address 127.0.97.3:44399)
0928 08:44:00.152739 (+ 3319us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:00.152764 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:00.152966 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:00.153474 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:00.153479 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:00.154189 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:00.154752 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:00.154768 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.156062 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.156068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:00.157631 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:00.157642 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.158664 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.158672 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:00.158870 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:00.159516 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:00.159532 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:00.160323 (+ 791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:00.163641 (+ 3318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:00.163644 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:00.163647 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:00.164154 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:00.164517 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:00.164520 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:00.164522 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:00.164706 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:00.165307 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:00.165310 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:00.165468 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 08:44:00.165566 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 08:44:00.165768 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3129,"thread_start_us":2909,"threads_started":1}
I20260928 08:44:00.166311 2382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:00.149414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44399 (local address 127.0.97.62:42255)
0928 08:44:00.149594 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:00.149601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:00.152853 (+ 3252us) server_negotiation.cc:407] Connection header received
0928 08:44:00.153108 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:00.153114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:00.153200 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:00.153308 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:00.154887 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.155933 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.157778 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.158208 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.158370 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:00.160523 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:00.160529 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:00.160533 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:00.160573 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:00.163539 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:00.164270 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:00.164275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:00.164278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:00.164414 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:00.164825 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:00.164832 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:00.164836 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:00.165188 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:00.165341 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:00.165702 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 08:44:00.165918 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 08:44:00.166515 2147 ts_manager.cc:194] Registered new tserver with Master: 469696e9fb2948138ebce9b1bd275d5c (127.0.97.3:37393)
I20260928 08:44:00.167853 2147 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33047
I20260928 08:44:00.169461 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 2498
I20260928 08:44:00.169641 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:44:00.170110 388 external_mini_cluster.cc:1428] Setting key a58c79cbbf009183c43b40efc70c73af
I20260928 08:44:00.172915 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 08:44:00 dist-test-slave-n6xs krb5kdc[2102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 08:44:00 dist-test-slave-n6xs krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:44:00.219332 2382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:00.205358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36972 (local address 127.0.97.62:42255)
0928 08:44:00.205608 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:00.205615 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:00.205888 (+ 273us) server_negotiation.cc:407] Connection header received
0928 08:44:00.206107 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:00.206111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:00.206180 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:00.206298 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:00.207625 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.208447 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.209698 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.210087 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.210221 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:00.214261 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:00.214266 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:00.214269 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:00.214299 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:00.216746 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:00.217359 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:00.217368 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:00.217374 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:00.217540 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:00.217943 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:00.217948 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:00.217951 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:00.218240 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:00.218484 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:00.218796 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 08:44:00.218996 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 08:44:00.223402 2147 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:44:00.230321 2147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:44:00.253686 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:00.243249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41732 (local address 127.0.97.1:45001)
0928 08:44:00.244056 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:00.244068 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:00.244451 (+ 383us) server_negotiation.cc:407] Connection header received
0928 08:44:00.244574 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:00.244582 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:00.244843 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:00.245050 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:00.246568 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.248027 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.250797 (+ 2770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.252142 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.252288 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:00.252499 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:00.253156 (+ 657us) server_negotiation.cc:299] Negotiation successful
0928 08:44:00.253339 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":379,"threads_started":1}
I20260928 08:44:00.253805 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:00.242653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45001 (local address 127.0.0.1:41732)
0928 08:44:00.244166 (+ 1513us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:00.244189 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:00.244319 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:00.245249 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:00.245253 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:00.245289 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:00.246304 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:00.246316 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.248197 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.248205 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:00.250609 (+ 2404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:00.250622 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.252633 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.252641 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:00.252873 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:00.252937 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 08:44:00.253088 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 08:44:00.253272 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1352,"mutex_wait_us":723,"spinlock_wait_cycles":896,"thread_start_us":1177,"threads_started":1}
I20260928 08:44:00.254415 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:00.242245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34607 (local address 127.0.0.1:44852)
0928 08:44:00.243294 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:00.243337 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:00.243508 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:00.245230 (+ 1722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:00.245238 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:00.245282 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:00.245790 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:00.245807 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.248225 (+ 2418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.248231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:00.251205 (+ 2974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:00.251220 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.253470 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.253478 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:00.253670 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:00.253716 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 08:44:00.253831 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 08:44:00.253975 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"mutex_wait_us":123,"spinlock_wait_cycles":1792,"thread_start_us":684,"threads_started":1}
I20260928 08:44:00.254571 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:00.244504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37393 (local address 127.0.0.1:53086)
0928 08:44:00.245097 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:00.245122 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:00.245250 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:00.247091 (+ 1841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:00.247096 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:00.247122 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:00.247471 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:00.247482 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.249505 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.249510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:00.251929 (+ 2419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:00.251944 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.253777 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.253790 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:00.253971 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:00.254005 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 08:44:00.254072 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 08:44:00.254195 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":3,"thread_start_us":860,"threads_started":1}
I20260928 08:44:00.255223 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:00.242510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44852 (local address 127.0.97.2:34607)
0928 08:44:00.244530 (+ 2020us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:00.244540 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:00.244563 (+ 23us) server_negotiation.cc:407] Connection header received
0928 08:44:00.244693 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:00.244700 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:00.244884 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:00.246330 (+ 1446us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:00.246543 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.248031 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.251380 (+ 3349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.252992 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.253116 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:00.253308 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:00.254740 (+ 1432us) server_negotiation.cc:299] Negotiation successful
0928 08:44:00.254915 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1852,"thread_start_us":1717,"threads_started":1}
I20260928 08:44:00.255656 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:00.245144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53086 (local address 127.0.97.3:37393)
0928 08:44:00.246469 (+ 1325us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:00.246477 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:00.246513 (+ 36us) server_negotiation.cc:407] Connection header received
0928 08:44:00.246652 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:00.246659 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:00.246888 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:00.247089 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:00.247678 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.249348 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.252121 (+ 2773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.253500 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.254907 (+ 1407us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:00.255100 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:00.255175 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 08:44:00.255339 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1160,"thread_start_us":898,"threads_started":1}
I20260928 08:44:00.257802 2284 tablet_service.cc:1467] Processing CreateTablet for tablet 15c51b5916484416b2d88755cb5f5d82 (DEFAULT_TABLE table=test-table [id=be7f7baba16d4b3fb8791b03fbf7284a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:00.258524 2424 tablet_service.cc:1467] Processing CreateTablet for tablet 15c51b5916484416b2d88755cb5f5d82 (DEFAULT_TABLE table=test-table [id=be7f7baba16d4b3fb8791b03fbf7284a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:00.258977 2563 tablet_service.cc:1467] Processing CreateTablet for tablet 15c51b5916484416b2d88755cb5f5d82 (DEFAULT_TABLE table=test-table [id=be7f7baba16d4b3fb8791b03fbf7284a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:00.259795 2284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15c51b5916484416b2d88755cb5f5d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:00.259920 2424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15c51b5916484416b2d88755cb5f5d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:00.260299 2563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15c51b5916484416b2d88755cb5f5d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:00.272426 2652 tablet_bootstrap.cc:492] T 15c51b5916484416b2d88755cb5f5d82 P 219436dd00ba4093adc396062d6ef10f: Bootstrap starting.
I20260928 08:44:00.274785 2654 tablet_bootstrap.cc:492] T 15c51b5916484416b2d88755cb5f5d82 P 469696e9fb2948138ebce9b1bd275d5c: Bootstrap starting.
I20260928 08:44:00.276420 2652 tablet_bootstrap.cc:654] T 15c51b5916484416b2d88755cb5f5d82 P 219436dd00ba4093adc396062d6ef10f: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:00.277839 2652 log.cc:824] T 15c51b5916484416b2d88755cb5f5d82 P 219436dd00ba4093adc396062d6ef10f: Log is configured to *not* fsync() on all Append() calls
I20260928 08:44:00.278473 2654 tablet_bootstrap.cc:654] T 15c51b5916484416b2d88755cb5f5d82 P 469696e9fb2948138ebce9b1bd275d5c: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:00.279189 2653 tablet_bootstrap.cc:492] T 15c51b5916484416b2d88755cb5f5d82 P 2a944f0f60ae43498dd4ef372b97458c: Bootstrap starting.
I20260928 08:44:00.279726 2654 log.cc:824] T 15c51b5916484416b2d88755cb5f5d82 P 469696e9fb2948138ebce9b1bd275d5c: Log is configured to *not* fsync() on all Append() calls
I20260928 08:44:00.282509 2653 tablet_bootstrap.cc:654] T 15c51b5916484416b2d88755cb5f5d82 P 2a944f0f60ae43498dd4ef372b97458c: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:00.282804 2654 tablet_bootstrap.cc:492] T 15c51b5916484416b2d88755cb5f5d82 P 469696e9fb2948138ebce9b1bd275d5c: No bootstrap required, opened a new log
I20260928 08:44:00.283103 2654 ts_tablet_manager.cc:1397] T 15c51b5916484416b2d88755cb5f5d82 P 469696e9fb2948138ebce9b1bd275d5c: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 08:44:00.283772 2653 log.cc:824] T 15c51b5916484416b2d88755cb5f5d82 P 2a944f0f60ae43498dd4ef372b97458c: Log is configured to *not* fsync() on all Append() calls
I20260928 08:44:00.284492 2652 tablet_bootstrap.cc:492] T 15c51b5916484416b2d88755cb5f5d82 P 219436dd00ba4093adc396062d6ef10f: No bootstrap required, opened a new log
I20260928 08:44:00.284762 2652 ts_tablet_manager.cc:1397] T 15c51b5916484416b2d88755cb5f5d82 P 219436dd00ba4093adc396062d6ef10f: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.004s
I20260928 08:44:00.286800 2653 tablet_bootstrap.cc:492] T 15c51b5916484416b2d88755cb5f5d82 P 2a944f0f60ae43498dd4ef372b97458c: No bootstrap required, opened a new log
I20260928 08:44:00.287055 2653 ts_tablet_manager.cc:1397] T 15c51b5916484416b2d88755cb5f5d82 P 2a944f0f60ae43498dd4ef372b97458c: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260928 08:44:00.292533 2652 raft_consensus.cc:348] T 15c51b5916484416b2d88755cb5f5d82 P 219436dd00ba4093adc396062d6ef10f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:00.292930 2652 raft_consensus.cc:374] T 15c51b5916484416b2d88755cb5f5d82 P 219436dd00ba4093adc396062d6ef10f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:00.293048 2652 raft_consensus.cc:729] T 15c51b5916484416b2d88755cb5f5d82 P 219436dd00ba4093adc396062d6ef10f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 219436dd00ba4093adc396062d6ef10f, State: Initialized, Role: FOLLOWER
I20260928 08:44:00.293699 2652 consensus_queue.cc:260] T 15c51b5916484416b2d88755cb5f5d82 P 219436dd00ba4093adc396062d6ef10f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:00.295367 2652 ts_tablet_manager.cc:1428] T 15c51b5916484416b2d88755cb5f5d82 P 219436dd00ba4093adc396062d6ef10f: Time spent starting tablet: real 0.010s user 0.008s sys 0.003s
I20260928 08:44:00.294579 2654 raft_consensus.cc:348] T 15c51b5916484416b2d88755cb5f5d82 P 469696e9fb2948138ebce9b1bd275d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:00.295957 2654 raft_consensus.cc:374] T 15c51b5916484416b2d88755cb5f5d82 P 469696e9fb2948138ebce9b1bd275d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:00.296114 2654 raft_consensus.cc:729] T 15c51b5916484416b2d88755cb5f5d82 P 469696e9fb2948138ebce9b1bd275d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 469696e9fb2948138ebce9b1bd275d5c, State: Initialized, Role: FOLLOWER
I20260928 08:44:00.296814 2654 consensus_queue.cc:260] T 15c51b5916484416b2d88755cb5f5d82 P 469696e9fb2948138ebce9b1bd275d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:00.298125 2653 raft_consensus.cc:348] T 15c51b5916484416b2d88755cb5f5d82 P 2a944f0f60ae43498dd4ef372b97458c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:00.298425 2653 raft_consensus.cc:374] T 15c51b5916484416b2d88755cb5f5d82 P 2a944f0f60ae43498dd4ef372b97458c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:00.298485 2653 raft_consensus.cc:729] T 15c51b5916484416b2d88755cb5f5d82 P 2a944f0f60ae43498dd4ef372b97458c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a944f0f60ae43498dd4ef372b97458c, State: Initialized, Role: FOLLOWER
I20260928 08:44:00.298971 2653 consensus_queue.cc:260] T 15c51b5916484416b2d88755cb5f5d82 P 2a944f0f60ae43498dd4ef372b97458c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:00.303251 2494 heartbeater.cc:503] Master 127.0.97.62:42255 was elected leader, sending a full tablet report...
I20260928 08:44:00.304109 2653 ts_tablet_manager.cc:1428] T 15c51b5916484416b2d88755cb5f5d82 P 2a944f0f60ae43498dd4ef372b97458c: Time spent starting tablet: real 0.017s user 0.013s sys 0.005s
I20260928 08:44:00.304575 2633 heartbeater.cc:503] Master 127.0.97.62:42255 was elected leader, sending a full tablet report...
I20260928 08:44:00.305299 2654 ts_tablet_manager.cc:1428] T 15c51b5916484416b2d88755cb5f5d82 P 469696e9fb2948138ebce9b1bd275d5c: Time spent starting tablet: real 0.022s user 0.012s sys 0.011s
W20260928 08:44:00.351922 2355 tablet.cc:2369] T 15c51b5916484416b2d88755cb5f5d82 P 219436dd00ba4093adc396062d6ef10f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:44:00.394379 2495 tablet.cc:2369] T 15c51b5916484416b2d88755cb5f5d82 P 2a944f0f60ae43498dd4ef372b97458c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:44:00.412560 2634 tablet.cc:2369] T 15c51b5916484416b2d88755cb5f5d82 P 469696e9fb2948138ebce9b1bd275d5c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:44:00.449060 2659 raft_consensus.cc:482] T 15c51b5916484416b2d88755cb5f5d82 P 469696e9fb2948138ebce9b1bd275d5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:44:00.449327 2659 raft_consensus.cc:504] T 15c51b5916484416b2d88755cb5f5d82 P 469696e9fb2948138ebce9b1bd275d5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:00.450896 2659 leader_election.cc:288] T 15c51b5916484416b2d88755cb5f5d82 P 469696e9fb2948138ebce9b1bd275d5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a944f0f60ae43498dd4ef372b97458c (127.0.97.2:34607), 219436dd00ba4093adc396062d6ef10f (127.0.97.1:45001)
I20260928 08:44:00.457602 2522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:00.451292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34607 (local address 127.0.97.3:47721)
0928 08:44:00.451745 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:00.451774 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:00.451971 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:00.452380 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:00.452385 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:00.452415 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:00.452634 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:00.452647 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.453884 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.453890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:00.455342 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:00.455353 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.456851 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.456857 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:00.457043 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:00.457087 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 08:44:00.457188 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 08:44:00.457318 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258}
I20260928 08:44:00.457631 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:00.451296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47721 (local address 127.0.97.2:34607)
0928 08:44:00.451523 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:00.451530 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:00.451887 (+ 357us) server_negotiation.cc:407] Connection header received
0928 08:44:00.452134 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:00.452143 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:00.452228 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:00.452325 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:00.452851 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.453753 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.455505 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.456670 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.456785 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:00.456869 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:00.457218 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 08:44:00.457314 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 08:44:00.458210 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:00.451415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53887 (local address 127.0.97.1:45001)
0928 08:44:00.451590 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:00.451594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:00.452225 (+ 631us) server_negotiation.cc:407] Connection header received
0928 08:44:00.452429 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:00.452435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:00.452511 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:00.452618 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:00.453310 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.454226 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.456304 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.457261 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.457378 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:00.457472 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:00.457915 (+ 443us) server_negotiation.cc:299] Negotiation successful
0928 08:44:00.458026 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 08:44:00.458746 2661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:00.451374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45001 (local address 127.0.97.3:53887)
0928 08:44:00.452100 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:00.452117 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:00.452256 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:00.452638 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:00.452641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:00.452658 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:00.453057 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:00.453067 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.454405 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.454409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:00.456128 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:00.456144 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.457499 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.457505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:00.457696 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:00.457739 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 08:44:00.458405 (+ 666us) client_negotiation.cc:240] Negotiation successful
0928 08:44:00.458508 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":380,"threads_started":1}
I20260928 08:44:00.458693 2444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c51b5916484416b2d88755cb5f5d82" candidate_uuid: "469696e9fb2948138ebce9b1bd275d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a944f0f60ae43498dd4ef372b97458c" is_pre_election: true
I20260928 08:44:00.459263 2444 raft_consensus.cc:2454] T 15c51b5916484416b2d88755cb5f5d82 P 2a944f0f60ae43498dd4ef372b97458c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 469696e9fb2948138ebce9b1bd275d5c in term 0.
I20260928 08:44:00.459776 2309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c51b5916484416b2d88755cb5f5d82" candidate_uuid: "469696e9fb2948138ebce9b1bd275d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "219436dd00ba4093adc396062d6ef10f" is_pre_election: true
I20260928 08:44:00.460028 2515 leader_election.cc:302] T 15c51b5916484416b2d88755cb5f5d82 P 469696e9fb2948138ebce9b1bd275d5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a944f0f60ae43498dd4ef372b97458c, 469696e9fb2948138ebce9b1bd275d5c; no voters:
I20260928 08:44:00.460321 2309 raft_consensus.cc:2454] T 15c51b5916484416b2d88755cb5f5d82 P 219436dd00ba4093adc396062d6ef10f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 469696e9fb2948138ebce9b1bd275d5c in term 0.
I20260928 08:44:00.460479 2659 raft_consensus.cc:2781] T 15c51b5916484416b2d88755cb5f5d82 P 469696e9fb2948138ebce9b1bd275d5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:44:00.460615 2659 raft_consensus.cc:482] T 15c51b5916484416b2d88755cb5f5d82 P 469696e9fb2948138ebce9b1bd275d5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:44:00.460723 2659 raft_consensus.cc:3037] T 15c51b5916484416b2d88755cb5f5d82 P 469696e9fb2948138ebce9b1bd275d5c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:00.462893 2659 raft_consensus.cc:504] T 15c51b5916484416b2d88755cb5f5d82 P 469696e9fb2948138ebce9b1bd275d5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:00.463454 2659 leader_election.cc:288] T 15c51b5916484416b2d88755cb5f5d82 P 469696e9fb2948138ebce9b1bd275d5c [CANDIDATE]: Term 1 election: Requested vote from peers 2a944f0f60ae43498dd4ef372b97458c (127.0.97.2:34607), 219436dd00ba4093adc396062d6ef10f (127.0.97.1:45001)
I20260928 08:44:00.463984 2309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c51b5916484416b2d88755cb5f5d82" candidate_uuid: "469696e9fb2948138ebce9b1bd275d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "219436dd00ba4093adc396062d6ef10f"
I20260928 08:44:00.463985 2444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15c51b5916484416b2d88755cb5f5d82" candidate_uuid: "469696e9fb2948138ebce9b1bd275d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a944f0f60ae43498dd4ef372b97458c"
I20260928 08:44:00.464181 2309 raft_consensus.cc:3037] T 15c51b5916484416b2d88755cb5f5d82 P 219436dd00ba4093adc396062d6ef10f [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:00.464181 2444 raft_consensus.cc:3037] T 15c51b5916484416b2d88755cb5f5d82 P 2a944f0f60ae43498dd4ef372b97458c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:00.467329 2309 raft_consensus.cc:2454] T 15c51b5916484416b2d88755cb5f5d82 P 219436dd00ba4093adc396062d6ef10f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 469696e9fb2948138ebce9b1bd275d5c in term 1.
I20260928 08:44:00.467335 2444 raft_consensus.cc:2454] T 15c51b5916484416b2d88755cb5f5d82 P 2a944f0f60ae43498dd4ef372b97458c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 469696e9fb2948138ebce9b1bd275d5c in term 1.
I20260928 08:44:00.467730 2515 leader_election.cc:302] T 15c51b5916484416b2d88755cb5f5d82 P 469696e9fb2948138ebce9b1bd275d5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a944f0f60ae43498dd4ef372b97458c, 469696e9fb2948138ebce9b1bd275d5c; no voters:
I20260928 08:44:00.468072 2659 raft_consensus.cc:2781] T 15c51b5916484416b2d88755cb5f5d82 P 469696e9fb2948138ebce9b1bd275d5c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:44:00.468498 2659 raft_consensus.cc:686] T 15c51b5916484416b2d88755cb5f5d82 P 469696e9fb2948138ebce9b1bd275d5c [term 1 LEADER]: Becoming Leader. State: Replica: 469696e9fb2948138ebce9b1bd275d5c, State: Running, Role: LEADER
I20260928 08:44:00.469007 2659 consensus_queue.cc:237] T 15c51b5916484416b2d88755cb5f5d82 P 469696e9fb2948138ebce9b1bd275d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:00.473572 2147 catalog_manager.cc:5495] T 15c51b5916484416b2d88755cb5f5d82 P 469696e9fb2948138ebce9b1bd275d5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 469696e9fb2948138ebce9b1bd275d5c (127.0.97.3). New cstate: current_term: 1 leader_uuid: "469696e9fb2948138ebce9b1bd275d5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } health_report { overall_health: HEALTHY } } }
Sep 28 08:44:00 dist-test-slave-n6xs krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:44:00.535596 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:00.522357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42255 (local address 127.0.0.1:36986)
0928 08:44:00.522962 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:00.522979 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:00.523111 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:00.523490 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:00.523494 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:00.524279 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:00.524530 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:00.524539 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.525547 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.525551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:00.526503 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:00.526513 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.527257 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.527262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:00.527377 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:00.528195 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:00.528220 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:00.530455 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:00.533337 (+ 2882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:00.533346 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:00.533351 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:00.533784 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:00.534144 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:00.534148 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:00.534150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:00.534289 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:00.534796 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:00.534803 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:00.534940 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 08:44:00.535079 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 08:44:00.535274 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":347,"threads_started":1}
I20260928 08:44:00.535728 2382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:00.522285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36986 (local address 127.0.97.62:42255)
0928 08:44:00.522485 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:00.522494 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:00.523079 (+ 585us) server_negotiation.cc:407] Connection header received
0928 08:44:00.523259 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:00.523262 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:00.523351 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:00.523436 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:00.524673 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.525394 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.526682 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.527082 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.527177 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:00.530627 (+ 3450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:00.530631 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:00.530634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:00.530660 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:00.533211 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:00.533888 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:00.533891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:00.533893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:00.534041 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:00.534378 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:00.534381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:00.534383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:00.534625 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:00.534756 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:00.535160 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 08:44:00.535449 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":66}
I20260928 08:44:00.540542 2147 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36986:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 08:44:00.541560 2147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:44:00.548350 2424 tablet_service.cc:1467] Processing CreateTablet for tablet 94e360f5718944b6b14c80081dfdc316 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c16d799ac19b4bcd9253d312dfb749a9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:44:00.548223 2284 tablet_service.cc:1467] Processing CreateTablet for tablet 94e360f5718944b6b14c80081dfdc316 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c16d799ac19b4bcd9253d312dfb749a9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:44:00.548692 2563 tablet_service.cc:1467] Processing CreateTablet for tablet 94e360f5718944b6b14c80081dfdc316 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c16d799ac19b4bcd9253d312dfb749a9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:44:00.549216 2424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e360f5718944b6b14c80081dfdc316. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:00.549230 2284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e360f5718944b6b14c80081dfdc316. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:00.549499 2563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94e360f5718944b6b14c80081dfdc316. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:00.554353 2654 tablet_bootstrap.cc:492] T 94e360f5718944b6b14c80081dfdc316 P 469696e9fb2948138ebce9b1bd275d5c: Bootstrap starting.
I20260928 08:44:00.555595 2654 tablet_bootstrap.cc:654] T 94e360f5718944b6b14c80081dfdc316 P 469696e9fb2948138ebce9b1bd275d5c: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:00.556106 2652 tablet_bootstrap.cc:492] T 94e360f5718944b6b14c80081dfdc316 P 219436dd00ba4093adc396062d6ef10f: Bootstrap starting.
I20260928 08:44:00.556123 2653 tablet_bootstrap.cc:492] T 94e360f5718944b6b14c80081dfdc316 P 2a944f0f60ae43498dd4ef372b97458c: Bootstrap starting.
I20260928 08:44:00.557665 2652 tablet_bootstrap.cc:654] T 94e360f5718944b6b14c80081dfdc316 P 219436dd00ba4093adc396062d6ef10f: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:00.557665 2653 tablet_bootstrap.cc:654] T 94e360f5718944b6b14c80081dfdc316 P 2a944f0f60ae43498dd4ef372b97458c: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:00.559083 2654 tablet_bootstrap.cc:492] T 94e360f5718944b6b14c80081dfdc316 P 469696e9fb2948138ebce9b1bd275d5c: No bootstrap required, opened a new log
I20260928 08:44:00.559221 2654 ts_tablet_manager.cc:1397] T 94e360f5718944b6b14c80081dfdc316 P 469696e9fb2948138ebce9b1bd275d5c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 08:44:00.560104 2654 raft_consensus.cc:348] T 94e360f5718944b6b14c80081dfdc316 P 469696e9fb2948138ebce9b1bd275d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:00.560365 2654 raft_consensus.cc:374] T 94e360f5718944b6b14c80081dfdc316 P 469696e9fb2948138ebce9b1bd275d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:00.560464 2654 raft_consensus.cc:729] T 94e360f5718944b6b14c80081dfdc316 P 469696e9fb2948138ebce9b1bd275d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 469696e9fb2948138ebce9b1bd275d5c, State: Initialized, Role: FOLLOWER
I20260928 08:44:00.560772 2652 tablet_bootstrap.cc:492] T 94e360f5718944b6b14c80081dfdc316 P 219436dd00ba4093adc396062d6ef10f: No bootstrap required, opened a new log
I20260928 08:44:00.560739 2654 consensus_queue.cc:260] T 94e360f5718944b6b14c80081dfdc316 P 469696e9fb2948138ebce9b1bd275d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:00.560916 2652 ts_tablet_manager.cc:1397] T 94e360f5718944b6b14c80081dfdc316 P 219436dd00ba4093adc396062d6ef10f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260928 08:44:00.561180 2659 tablet_replica.cc:439] T 94e360f5718944b6b14c80081dfdc316 P 469696e9fb2948138ebce9b1bd275d5c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } } }
I20260928 08:44:00.561465 2659 tablet_replica.cc:442] T 94e360f5718944b6b14c80081dfdc316 P 469696e9fb2948138ebce9b1bd275d5c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:44:00.561622 2653 tablet_bootstrap.cc:492] T 94e360f5718944b6b14c80081dfdc316 P 2a944f0f60ae43498dd4ef372b97458c: No bootstrap required, opened a new log
I20260928 08:44:00.561771 2653 ts_tablet_manager.cc:1397] T 94e360f5718944b6b14c80081dfdc316 P 2a944f0f60ae43498dd4ef372b97458c: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260928 08:44:00.561630 2652 raft_consensus.cc:348] T 94e360f5718944b6b14c80081dfdc316 P 219436dd00ba4093adc396062d6ef10f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:00.561921 2652 raft_consensus.cc:374] T 94e360f5718944b6b14c80081dfdc316 P 219436dd00ba4093adc396062d6ef10f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:00.562019 2652 raft_consensus.cc:729] T 94e360f5718944b6b14c80081dfdc316 P 219436dd00ba4093adc396062d6ef10f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 219436dd00ba4093adc396062d6ef10f, State: Initialized, Role: FOLLOWER
I20260928 08:44:00.562333 2654 ts_tablet_manager.cc:1428] T 94e360f5718944b6b14c80081dfdc316 P 469696e9fb2948138ebce9b1bd275d5c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 08:44:00.562282 2652 consensus_queue.cc:260] T 94e360f5718944b6b14c80081dfdc316 P 219436dd00ba4093adc396062d6ef10f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:00.562592 2653 raft_consensus.cc:348] T 94e360f5718944b6b14c80081dfdc316 P 2a944f0f60ae43498dd4ef372b97458c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:00.562825 2653 raft_consensus.cc:374] T 94e360f5718944b6b14c80081dfdc316 P 2a944f0f60ae43498dd4ef372b97458c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:00.562907 2653 raft_consensus.cc:729] T 94e360f5718944b6b14c80081dfdc316 P 2a944f0f60ae43498dd4ef372b97458c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a944f0f60ae43498dd4ef372b97458c, State: Initialized, Role: FOLLOWER
I20260928 08:44:00.562783 2658 tablet_replica.cc:439] T 94e360f5718944b6b14c80081dfdc316 P 219436dd00ba4093adc396062d6ef10f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } } }
I20260928 08:44:00.563172 2658 tablet_replica.cc:442] T 94e360f5718944b6b14c80081dfdc316 P 219436dd00ba4093adc396062d6ef10f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:44:00.563148 2653 consensus_queue.cc:260] T 94e360f5718944b6b14c80081dfdc316 P 2a944f0f60ae43498dd4ef372b97458c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:00.563604 2652 ts_tablet_manager.cc:1428] T 94e360f5718944b6b14c80081dfdc316 P 219436dd00ba4093adc396062d6ef10f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 08:44:00.563728 2660 tablet_replica.cc:439] T 94e360f5718944b6b14c80081dfdc316 P 2a944f0f60ae43498dd4ef372b97458c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } } }
I20260928 08:44:00.564064 2660 tablet_replica.cc:442] T 94e360f5718944b6b14c80081dfdc316 P 2a944f0f60ae43498dd4ef372b97458c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:44:00.564814 2653 ts_tablet_manager.cc:1428] T 94e360f5718944b6b14c80081dfdc316 P 2a944f0f60ae43498dd4ef372b97458c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 08:44:00.596700 2660 raft_consensus.cc:482] T 94e360f5718944b6b14c80081dfdc316 P 2a944f0f60ae43498dd4ef372b97458c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:44:00.596961 2660 raft_consensus.cc:504] T 94e360f5718944b6b14c80081dfdc316 P 2a944f0f60ae43498dd4ef372b97458c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:00.598472 2660 leader_election.cc:288] T 94e360f5718944b6b14c80081dfdc316 P 2a944f0f60ae43498dd4ef372b97458c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 219436dd00ba4093adc396062d6ef10f (127.0.97.1:45001), 469696e9fb2948138ebce9b1bd275d5c (127.0.97.3:37393)
I20260928 08:44:00.607045 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:00.599762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47319 (local address 127.0.97.3:37393)
0928 08:44:00.599983 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:00.599990 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:00.600345 (+ 355us) server_negotiation.cc:407] Connection header received
0928 08:44:00.600519 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:00.600523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:00.600577 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:00.600645 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:00.601212 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.602214 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.604820 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.605972 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.606081 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:00.606195 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:00.606650 (+ 455us) server_negotiation.cc:299] Negotiation successful
0928 08:44:00.606778 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 08:44:00.607069 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:00.598912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41875 (local address 127.0.97.1:45001)
0928 08:44:00.599084 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:00.599089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:00.599598 (+ 509us) server_negotiation.cc:407] Connection header received
0928 08:44:00.599904 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:00.599914 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:00.600014 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:00.600133 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:00.600783 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.601784 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.604623 (+ 2839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.605901 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.606047 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:00.606158 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:00.606650 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 08:44:00.606808 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 08:44:00.607226 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:00.599639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37393 (local address 127.0.97.2:47319)
0928 08:44:00.600225 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:00.600240 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:00.600386 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:00.600683 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:00.600689 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:00.600718 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:00.601042 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:00.601050 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.602359 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.602364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:00.604671 (+ 2307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:00.604680 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.606213 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.606217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:00.606457 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:00.606514 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 08:44:00.606609 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 08:44:00.606788 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"mutex_wait_us":84,"thread_start_us":283,"threads_started":1}
I20260928 08:44:00.607357 2673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:00.598816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45001 (local address 127.0.97.2:41875)
0928 08:44:00.599482 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:00.599505 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:00.599671 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:00.600183 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:00.600188 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:00.600226 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:00.600657 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:00.600669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.601923 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.601928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:00.604441 (+ 2513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:00.604455 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.606203 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.606210 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:00.606472 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:00.606521 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 08:44:00.606609 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 08:44:00.606785 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":79,"thread_start_us":429,"threads_started":1}
I20260928 08:44:00.608098 2309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e360f5718944b6b14c80081dfdc316" candidate_uuid: "2a944f0f60ae43498dd4ef372b97458c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "219436dd00ba4093adc396062d6ef10f" is_pre_election: true
I20260928 08:44:00.608166 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e360f5718944b6b14c80081dfdc316" candidate_uuid: "2a944f0f60ae43498dd4ef372b97458c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "469696e9fb2948138ebce9b1bd275d5c" is_pre_election: true
I20260928 08:44:00.608413 2309 raft_consensus.cc:2454] T 94e360f5718944b6b14c80081dfdc316 P 219436dd00ba4093adc396062d6ef10f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a944f0f60ae43498dd4ef372b97458c in term 0.
I20260928 08:44:00.608583 2583 raft_consensus.cc:2454] T 94e360f5718944b6b14c80081dfdc316 P 469696e9fb2948138ebce9b1bd275d5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a944f0f60ae43498dd4ef372b97458c in term 0.
I20260928 08:44:00.608915 2374 leader_election.cc:302] T 94e360f5718944b6b14c80081dfdc316 P 2a944f0f60ae43498dd4ef372b97458c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 219436dd00ba4093adc396062d6ef10f, 2a944f0f60ae43498dd4ef372b97458c; no voters:
I20260928 08:44:00.609306 2660 raft_consensus.cc:2781] T 94e360f5718944b6b14c80081dfdc316 P 2a944f0f60ae43498dd4ef372b97458c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:44:00.609431 2660 raft_consensus.cc:482] T 94e360f5718944b6b14c80081dfdc316 P 2a944f0f60ae43498dd4ef372b97458c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:44:00.609526 2660 raft_consensus.cc:3037] T 94e360f5718944b6b14c80081dfdc316 P 2a944f0f60ae43498dd4ef372b97458c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:00.611567 2660 raft_consensus.cc:504] T 94e360f5718944b6b14c80081dfdc316 P 2a944f0f60ae43498dd4ef372b97458c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:00.612246 2660 leader_election.cc:288] T 94e360f5718944b6b14c80081dfdc316 P 2a944f0f60ae43498dd4ef372b97458c [CANDIDATE]: Term 1 election: Requested vote from peers 219436dd00ba4093adc396062d6ef10f (127.0.97.1:45001), 469696e9fb2948138ebce9b1bd275d5c (127.0.97.3:37393)
I20260928 08:44:00.612617 2309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e360f5718944b6b14c80081dfdc316" candidate_uuid: "2a944f0f60ae43498dd4ef372b97458c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "219436dd00ba4093adc396062d6ef10f"
I20260928 08:44:00.612738 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94e360f5718944b6b14c80081dfdc316" candidate_uuid: "2a944f0f60ae43498dd4ef372b97458c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "469696e9fb2948138ebce9b1bd275d5c"
I20260928 08:44:00.612830 2309 raft_consensus.cc:3037] T 94e360f5718944b6b14c80081dfdc316 P 219436dd00ba4093adc396062d6ef10f [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:00.612957 2583 raft_consensus.cc:3037] T 94e360f5718944b6b14c80081dfdc316 P 469696e9fb2948138ebce9b1bd275d5c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:00.615173 2309 raft_consensus.cc:2454] T 94e360f5718944b6b14c80081dfdc316 P 219436dd00ba4093adc396062d6ef10f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a944f0f60ae43498dd4ef372b97458c in term 1.
I20260928 08:44:00.615487 2583 raft_consensus.cc:2454] T 94e360f5718944b6b14c80081dfdc316 P 469696e9fb2948138ebce9b1bd275d5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a944f0f60ae43498dd4ef372b97458c in term 1.
I20260928 08:44:00.615532 2374 leader_election.cc:302] T 94e360f5718944b6b14c80081dfdc316 P 2a944f0f60ae43498dd4ef372b97458c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 219436dd00ba4093adc396062d6ef10f, 2a944f0f60ae43498dd4ef372b97458c; no voters:
I20260928 08:44:00.615795 2660 raft_consensus.cc:2781] T 94e360f5718944b6b14c80081dfdc316 P 2a944f0f60ae43498dd4ef372b97458c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:44:00.616248 2660 raft_consensus.cc:686] T 94e360f5718944b6b14c80081dfdc316 P 2a944f0f60ae43498dd4ef372b97458c [term 1 LEADER]: Becoming Leader. State: Replica: 2a944f0f60ae43498dd4ef372b97458c, State: Running, Role: LEADER
I20260928 08:44:00.616783 2660 consensus_queue.cc:237] T 94e360f5718944b6b14c80081dfdc316 P 2a944f0f60ae43498dd4ef372b97458c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:00.620471 2675 tablet_replica.cc:439] T 94e360f5718944b6b14c80081dfdc316 P 2a944f0f60ae43498dd4ef372b97458c: TxnStatusTablet state changed. Reason: New leader 2a944f0f60ae43498dd4ef372b97458c. Latest consensus state: current_term: 1 leader_uuid: "2a944f0f60ae43498dd4ef372b97458c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } } }
I20260928 08:44:00.620882 2675 tablet_replica.cc:442] T 94e360f5718944b6b14c80081dfdc316 P 2a944f0f60ae43498dd4ef372b97458c: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:44:00.621594 2677 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 08:44:00.621893 2677 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 08:44:00.621850 2145 catalog_manager.cc:5495] T 94e360f5718944b6b14c80081dfdc316 P 2a944f0f60ae43498dd4ef372b97458c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a944f0f60ae43498dd4ef372b97458c (127.0.97.2). New cstate: current_term: 1 leader_uuid: "2a944f0f60ae43498dd4ef372b97458c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } health_report { overall_health: UNKNOWN } } }
I20260928 08:44:00.646486 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:00.638800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44860 (local address 127.0.97.2:34607)
0928 08:44:00.638997 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:00.639003 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:00.639322 (+ 319us) server_negotiation.cc:407] Connection header received
0928 08:44:00.639640 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:00.639646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:00.639763 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:00.639921 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:44:00.640336 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.641594 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.643685 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.644386 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.644541 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:00.645173 (+ 632us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:44:00.645810 (+ 637us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:44:00.645967 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:00.646100 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 08:44:00.646183 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 08:44:00.646571 2669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:00.638808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34607 (local address 127.0.0.1:44860)
0928 08:44:00.639212 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:00.639236 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:00.639395 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:00.639964 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:00.639971 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:00.639986 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:44:00.640200 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:00.640213 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.641750 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.641757 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:00.643536 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:00.643552 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.644570 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.644576 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:00.644817 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:00.644889 (+ 72us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:44:00.645925 (+ 1036us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:44:00.645948 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 08:44:00.646083 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 08:44:00.646235 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226}
I20260928 08:44:00.656210 2583 raft_consensus.cc:1261] T 94e360f5718944b6b14c80081dfdc316 P 469696e9fb2948138ebce9b1bd275d5c [term 1 FOLLOWER]: Refusing update from remote peer 2a944f0f60ae43498dd4ef372b97458c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:44:00.656292 2309 raft_consensus.cc:1261] T 94e360f5718944b6b14c80081dfdc316 P 219436dd00ba4093adc396062d6ef10f [term 1 FOLLOWER]: Refusing update from remote peer 2a944f0f60ae43498dd4ef372b97458c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:44:00.657318 2675 consensus_queue.cc:1035] T 94e360f5718944b6b14c80081dfdc316 P 2a944f0f60ae43498dd4ef372b97458c [LEADER]: Connected to new peer: Peer: permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:44:00.657778 2660 consensus_queue.cc:1035] T 94e360f5718944b6b14c80081dfdc316 P 2a944f0f60ae43498dd4ef372b97458c [LEADER]: Connected to new peer: Peer: permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:44:00.666280 2658 tablet_replica.cc:439] T 94e360f5718944b6b14c80081dfdc316 P 219436dd00ba4093adc396062d6ef10f: TxnStatusTablet state changed. Reason: New leader 2a944f0f60ae43498dd4ef372b97458c. Latest consensus state: current_term: 1 leader_uuid: "2a944f0f60ae43498dd4ef372b97458c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } } }
I20260928 08:44:00.666569 2658 tablet_replica.cc:442] T 94e360f5718944b6b14c80081dfdc316 P 219436dd00ba4093adc396062d6ef10f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:44:00.670744 2659 tablet_replica.cc:439] T 94e360f5718944b6b14c80081dfdc316 P 469696e9fb2948138ebce9b1bd275d5c: TxnStatusTablet state changed. Reason: New leader 2a944f0f60ae43498dd4ef372b97458c. Latest consensus state: current_term: 1 leader_uuid: "2a944f0f60ae43498dd4ef372b97458c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } } }
I20260928 08:44:00.671072 2659 tablet_replica.cc:442] T 94e360f5718944b6b14c80081dfdc316 P 469696e9fb2948138ebce9b1bd275d5c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:44:00.671646 2680 mvcc.cc:205] Tried to move back new op lower bound from 7334236326517448704 to 7334236326370435072. Current Snapshot: MvccSnapshot[applied={T|T < 7334236326517448704}]
I20260928 08:44:00.672654 2681 mvcc.cc:205] Tried to move back new op lower bound from 7334236326517448704 to 7334236326370435072. Current Snapshot: MvccSnapshot[applied={T|T < 7334236326517448704}]
I20260928 08:44:00.673823 2682 mvcc.cc:205] Tried to move back new op lower bound from 7334236326517448704 to 7334236326370435072. Current Snapshot: MvccSnapshot[applied={T|T < 7334236326517448704}]
I20260928 08:44:00.680282 2659 tablet_replica.cc:439] T 94e360f5718944b6b14c80081dfdc316 P 469696e9fb2948138ebce9b1bd275d5c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a944f0f60ae43498dd4ef372b97458c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } } }
I20260928 08:44:00.680547 2659 tablet_replica.cc:442] T 94e360f5718944b6b14c80081dfdc316 P 469696e9fb2948138ebce9b1bd275d5c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:44:00.683367 2658 tablet_replica.cc:439] T 94e360f5718944b6b14c80081dfdc316 P 219436dd00ba4093adc396062d6ef10f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a944f0f60ae43498dd4ef372b97458c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } } }
I20260928 08:44:00.683614 2658 tablet_replica.cc:442] T 94e360f5718944b6b14c80081dfdc316 P 219436dd00ba4093adc396062d6ef10f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:44:00.687415 2675 tablet_replica.cc:439] T 94e360f5718944b6b14c80081dfdc316 P 2a944f0f60ae43498dd4ef372b97458c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a944f0f60ae43498dd4ef372b97458c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } } }
I20260928 08:44:00.687686 2675 tablet_replica.cc:442] T 94e360f5718944b6b14c80081dfdc316 P 2a944f0f60ae43498dd4ef372b97458c: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:44:00.688110 2660 tablet_replica.cc:439] T 94e360f5718944b6b14c80081dfdc316 P 2a944f0f60ae43498dd4ef372b97458c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a944f0f60ae43498dd4ef372b97458c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } } }
I20260928 08:44:00.688432 2660 tablet_replica.cc:442] T 94e360f5718944b6b14c80081dfdc316 P 2a944f0f60ae43498dd4ef372b97458c: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:44:00.699928 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:00.693397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53098 (local address 127.0.97.3:37393)
0928 08:44:00.693577 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:00.693584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:00.693717 (+ 133us) server_negotiation.cc:407] Connection header received
0928 08:44:00.693923 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:00.693929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:00.694047 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:00.694222 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:44:00.694546 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.695908 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.697321 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.697984 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.698118 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:00.698533 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:44:00.699268 (+ 735us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:44:00.699406 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:00.699490 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 08:44:00.699577 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 08:44:00.717381 2692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:00.710773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34607 (local address 127.0.97.3:42903)
0928 08:44:00.711518 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:00.711536 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:00.711662 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:00.712052 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:00.712055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:00.712065 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:44:00.712458 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:00.712471 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.714110 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.714115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:00.715223 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:00.715233 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.716330 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.716334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:00.716455 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:00.716502 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:44:00.716950 (+ 448us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:44:00.716958 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 08:44:00.717022 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 08:44:00.717112 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":390,"threads_started":1}
I20260928 08:44:00.717478 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:00.710816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42903 (local address 127.0.97.2:34607)
0928 08:44:00.711015 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:00.711022 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:00.711622 (+ 600us) server_negotiation.cc:407] Connection header received
0928 08:44:00.711788 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:00.711795 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:00.711927 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:00.712037 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:44:00.712654 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.713964 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.715363 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.716156 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.716300 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:00.716703 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:44:00.716831 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:44:00.716955 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:00.717075 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 08:44:00.717168 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 08:44:00.727236 2309 raft_consensus.cc:1261] T 15c51b5916484416b2d88755cb5f5d82 P 219436dd00ba4093adc396062d6ef10f [term 1 FOLLOWER]: Refusing update from remote peer 469696e9fb2948138ebce9b1bd275d5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:44:00.727550 2444 raft_consensus.cc:1261] T 15c51b5916484416b2d88755cb5f5d82 P 2a944f0f60ae43498dd4ef372b97458c [term 1 FOLLOWER]: Refusing update from remote peer 469696e9fb2948138ebce9b1bd275d5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:44:00.727965 2662 consensus_queue.cc:1035] T 15c51b5916484416b2d88755cb5f5d82 P 469696e9fb2948138ebce9b1bd275d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:44:00.728369 2659 consensus_queue.cc:1035] T 15c51b5916484416b2d88755cb5f5d82 P 469696e9fb2948138ebce9b1bd275d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:44:00.772233 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:00.766172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39579 (local address 127.0.97.3:37393)
0928 08:44:00.766355 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:00.766359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:00.766532 (+ 173us) server_negotiation.cc:407] Connection header received
0928 08:44:00.766758 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:00.766761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:00.766821 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:00.766893 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:44:00.767611 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.768447 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.770389 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.770826 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.770959 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:00.771405 (+ 446us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:44:00.771497 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:44:00.771591 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:00.771888 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 08:44:00.771958 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 08:44:00.772306 2698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:00.765628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37393 (local address 127.0.97.2:39579)
0928 08:44:00.766426 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:00.766451 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:00.766604 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:00.766947 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:00.766953 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:00.766967 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:44:00.767396 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:00.767409 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.768577 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.768585 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:00.770238 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:00.770253 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.771017 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.771023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:00.771169 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:00.771221 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:44:00.771593 (+ 372us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:44:00.771613 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 08:44:00.771695 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 08:44:00.771872 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":419,"threads_started":1}
I20260928 08:44:00.809013 2147 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:44:00.809303 2147 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:44:00.813314 2147 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 353e51706cce4c289e9293253cc0d1dd: Sending DeleteTablet for 3 replicas of tablet 15c51b5916484416b2d88755cb5f5d82
I20260928 08:44:00.814460 2424 tablet_service.cc:1514] Processing DeleteTablet for tablet 15c51b5916484416b2d88755cb5f5d82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44852
I20260928 08:44:00.814963 2284 tablet_service.cc:1514] Processing DeleteTablet for tablet 15c51b5916484416b2d88755cb5f5d82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41732
I20260928 08:44:00.817058 2147 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 08:44:00.817243 2563 tablet_service.cc:1514] Processing DeleteTablet for tablet 15c51b5916484416b2d88755cb5f5d82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53086
W20260928 08:44:00.817801 2147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:44:00.826330 2423 tablet_service.cc:1467] Processing CreateTablet for tablet 154579534b604b038d04380e3f5b94dc (DEFAULT_TABLE table=test-table [id=1d24c60ce17644709fb3fff93f74e5d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:00.826797 2562 tablet_service.cc:1467] Processing CreateTablet for tablet 154579534b604b038d04380e3f5b94dc (DEFAULT_TABLE table=test-table [id=1d24c60ce17644709fb3fff93f74e5d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:00.827000 2423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 154579534b604b038d04380e3f5b94dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:00.827399 2562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 154579534b604b038d04380e3f5b94dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:00.832263 2283 tablet_service.cc:1467] Processing CreateTablet for tablet 154579534b604b038d04380e3f5b94dc (DEFAULT_TABLE table=test-table [id=1d24c60ce17644709fb3fff93f74e5d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:00.832753 2283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 154579534b604b038d04380e3f5b94dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:00.834987 2703 tablet_replica.cc:330] T 15c51b5916484416b2d88755cb5f5d82 P 469696e9fb2948138ebce9b1bd275d5c: stopping tablet replica
I20260928 08:44:00.835089 2654 tablet_bootstrap.cc:492] T 154579534b604b038d04380e3f5b94dc P 469696e9fb2948138ebce9b1bd275d5c: Bootstrap starting.
I20260928 08:44:00.835424 2703 raft_consensus.cc:2229] T 15c51b5916484416b2d88755cb5f5d82 P 469696e9fb2948138ebce9b1bd275d5c [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:44:00.836081 2703 raft_consensus.cc:2258] T 15c51b5916484416b2d88755cb5f5d82 P 469696e9fb2948138ebce9b1bd275d5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:00.837117 2654 tablet_bootstrap.cc:654] T 154579534b604b038d04380e3f5b94dc P 469696e9fb2948138ebce9b1bd275d5c: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:00.838129 2703 ts_tablet_manager.cc:1905] T 15c51b5916484416b2d88755cb5f5d82 P 469696e9fb2948138ebce9b1bd275d5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:00.843072 2652 tablet_bootstrap.cc:492] T 154579534b604b038d04380e3f5b94dc P 219436dd00ba4093adc396062d6ef10f: Bootstrap starting.
I20260928 08:44:00.844760 2652 tablet_bootstrap.cc:654] T 154579534b604b038d04380e3f5b94dc P 219436dd00ba4093adc396062d6ef10f: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:00.854977 2702 tablet_replica.cc:330] T 15c51b5916484416b2d88755cb5f5d82 P 219436dd00ba4093adc396062d6ef10f: stopping tablet replica
I20260928 08:44:00.855010 2653 tablet_bootstrap.cc:492] T 154579534b604b038d04380e3f5b94dc P 2a944f0f60ae43498dd4ef372b97458c: Bootstrap starting.
I20260928 08:44:00.855886 2654 tablet_bootstrap.cc:492] T 154579534b604b038d04380e3f5b94dc P 469696e9fb2948138ebce9b1bd275d5c: No bootstrap required, opened a new log
I20260928 08:44:00.856835 2653 tablet_bootstrap.cc:654] T 154579534b604b038d04380e3f5b94dc P 2a944f0f60ae43498dd4ef372b97458c: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:00.857228 2654 ts_tablet_manager.cc:1397] T 154579534b604b038d04380e3f5b94dc P 469696e9fb2948138ebce9b1bd275d5c: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.009s
I20260928 08:44:00.858722 2654 raft_consensus.cc:348] T 154579534b604b038d04380e3f5b94dc P 469696e9fb2948138ebce9b1bd275d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:00.859095 2654 raft_consensus.cc:374] T 154579534b604b038d04380e3f5b94dc P 469696e9fb2948138ebce9b1bd275d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:00.859239 2654 raft_consensus.cc:729] T 154579534b604b038d04380e3f5b94dc P 469696e9fb2948138ebce9b1bd275d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 469696e9fb2948138ebce9b1bd275d5c, State: Initialized, Role: FOLLOWER
I20260928 08:44:00.859642 2654 consensus_queue.cc:260] T 154579534b604b038d04380e3f5b94dc P 469696e9fb2948138ebce9b1bd275d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:00.861428 2703 ts_tablet_manager.cc:1918] T 15c51b5916484416b2d88755cb5f5d82 P 469696e9fb2948138ebce9b1bd275d5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:44:00.861604 2703 log.cc:1196] T 15c51b5916484416b2d88755cb5f5d82 P 469696e9fb2948138ebce9b1bd275d5c: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-2/wal/wals/15c51b5916484416b2d88755cb5f5d82
I20260928 08:44:00.862180 2703 ts_tablet_manager.cc:1939] T 15c51b5916484416b2d88755cb5f5d82 P 469696e9fb2948138ebce9b1bd275d5c: Deleting consensus metadata
I20260928 08:44:00.864089 2654 ts_tablet_manager.cc:1428] T 154579534b604b038d04380e3f5b94dc P 469696e9fb2948138ebce9b1bd275d5c: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260928 08:44:00.864065 2131 catalog_manager.cc:4841] TS 469696e9fb2948138ebce9b1bd275d5c (127.0.97.3:37393): tablet 15c51b5916484416b2d88755cb5f5d82 (table test-table [id=be7f7baba16d4b3fb8791b03fbf7284a]) successfully deleted
I20260928 08:44:00.867194 2702 raft_consensus.cc:2229] T 15c51b5916484416b2d88755cb5f5d82 P 219436dd00ba4093adc396062d6ef10f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:44:00.867682 2702 raft_consensus.cc:2258] T 15c51b5916484416b2d88755cb5f5d82 P 219436dd00ba4093adc396062d6ef10f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:00.870860 2702 ts_tablet_manager.cc:1905] T 15c51b5916484416b2d88755cb5f5d82 P 219436dd00ba4093adc396062d6ef10f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:00.872656 2653 tablet_bootstrap.cc:492] T 154579534b604b038d04380e3f5b94dc P 2a944f0f60ae43498dd4ef372b97458c: No bootstrap required, opened a new log
I20260928 08:44:00.872897 2653 ts_tablet_manager.cc:1397] T 154579534b604b038d04380e3f5b94dc P 2a944f0f60ae43498dd4ef372b97458c: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.009s
I20260928 08:44:00.873837 2653 raft_consensus.cc:348] T 154579534b604b038d04380e3f5b94dc P 2a944f0f60ae43498dd4ef372b97458c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:00.874178 2653 raft_consensus.cc:374] T 154579534b604b038d04380e3f5b94dc P 2a944f0f60ae43498dd4ef372b97458c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:00.874329 2653 raft_consensus.cc:729] T 154579534b604b038d04380e3f5b94dc P 2a944f0f60ae43498dd4ef372b97458c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a944f0f60ae43498dd4ef372b97458c, State: Initialized, Role: FOLLOWER
I20260928 08:44:00.874679 2653 consensus_queue.cc:260] T 154579534b604b038d04380e3f5b94dc P 2a944f0f60ae43498dd4ef372b97458c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:00.875494 2653 ts_tablet_manager.cc:1428] T 154579534b604b038d04380e3f5b94dc P 2a944f0f60ae43498dd4ef372b97458c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 08:44:00.873777 2652 tablet_bootstrap.cc:492] T 154579534b604b038d04380e3f5b94dc P 219436dd00ba4093adc396062d6ef10f: No bootstrap required, opened a new log
I20260928 08:44:00.879583 2652 ts_tablet_manager.cc:1397] T 154579534b604b038d04380e3f5b94dc P 219436dd00ba4093adc396062d6ef10f: Time spent bootstrapping tablet: real 0.037s user 0.005s sys 0.007s
I20260928 08:44:00.880440 2652 raft_consensus.cc:348] T 154579534b604b038d04380e3f5b94dc P 219436dd00ba4093adc396062d6ef10f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:00.880681 2652 raft_consensus.cc:374] T 154579534b604b038d04380e3f5b94dc P 219436dd00ba4093adc396062d6ef10f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:00.880764 2652 raft_consensus.cc:729] T 154579534b604b038d04380e3f5b94dc P 219436dd00ba4093adc396062d6ef10f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 219436dd00ba4093adc396062d6ef10f, State: Initialized, Role: FOLLOWER
I20260928 08:44:00.881093 2652 consensus_queue.cc:260] T 154579534b604b038d04380e3f5b94dc P 219436dd00ba4093adc396062d6ef10f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:00.883751 2652 ts_tablet_manager.cc:1428] T 154579534b604b038d04380e3f5b94dc P 219436dd00ba4093adc396062d6ef10f: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260928 08:44:00.886435 2702 ts_tablet_manager.cc:1918] T 15c51b5916484416b2d88755cb5f5d82 P 219436dd00ba4093adc396062d6ef10f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:44:00.886650 2702 log.cc:1196] T 15c51b5916484416b2d88755cb5f5d82 P 219436dd00ba4093adc396062d6ef10f: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-0/wal/wals/15c51b5916484416b2d88755cb5f5d82
I20260928 08:44:00.887274 2702 ts_tablet_manager.cc:1939] T 15c51b5916484416b2d88755cb5f5d82 P 219436dd00ba4093adc396062d6ef10f: Deleting consensus metadata
I20260928 08:44:00.888295 2701 tablet_replica.cc:330] T 15c51b5916484416b2d88755cb5f5d82 P 2a944f0f60ae43498dd4ef372b97458c: stopping tablet replica
I20260928 08:44:00.888732 2701 raft_consensus.cc:2229] T 15c51b5916484416b2d88755cb5f5d82 P 2a944f0f60ae43498dd4ef372b97458c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:44:00.889067 2701 raft_consensus.cc:2258] T 15c51b5916484416b2d88755cb5f5d82 P 2a944f0f60ae43498dd4ef372b97458c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:00.889492 2131 catalog_manager.cc:4841] TS 219436dd00ba4093adc396062d6ef10f (127.0.97.1:45001): tablet 15c51b5916484416b2d88755cb5f5d82 (table test-table [id=be7f7baba16d4b3fb8791b03fbf7284a]) successfully deleted
I20260928 08:44:00.891242 2701 ts_tablet_manager.cc:1905] T 15c51b5916484416b2d88755cb5f5d82 P 2a944f0f60ae43498dd4ef372b97458c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:00.899217 2701 ts_tablet_manager.cc:1918] T 15c51b5916484416b2d88755cb5f5d82 P 2a944f0f60ae43498dd4ef372b97458c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:44:00.899382 2701 log.cc:1196] T 15c51b5916484416b2d88755cb5f5d82 P 2a944f0f60ae43498dd4ef372b97458c: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-1/wal/wals/15c51b5916484416b2d88755cb5f5d82
I20260928 08:44:00.899928 2701 ts_tablet_manager.cc:1939] T 15c51b5916484416b2d88755cb5f5d82 P 2a944f0f60ae43498dd4ef372b97458c: Deleting consensus metadata
I20260928 08:44:00.900420 2658 raft_consensus.cc:482] T 154579534b604b038d04380e3f5b94dc P 219436dd00ba4093adc396062d6ef10f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:44:00.900635 2658 raft_consensus.cc:504] T 154579534b604b038d04380e3f5b94dc P 219436dd00ba4093adc396062d6ef10f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:00.901722 2132 catalog_manager.cc:4841] TS 2a944f0f60ae43498dd4ef372b97458c (127.0.97.2:34607): tablet 15c51b5916484416b2d88755cb5f5d82 (table test-table [id=be7f7baba16d4b3fb8791b03fbf7284a]) successfully deleted
I20260928 08:44:00.902596 2658 leader_election.cc:288] T 154579534b604b038d04380e3f5b94dc P 219436dd00ba4093adc396062d6ef10f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a944f0f60ae43498dd4ef372b97458c (127.0.97.2:34607), 469696e9fb2948138ebce9b1bd275d5c (127.0.97.3:37393)
I20260928 08:44:00.922019 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:00.903350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33635 (local address 127.0.97.2:34607)
0928 08:44:00.903557 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:00.903564 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:00.913904 (+ 10340us) server_negotiation.cc:407] Connection header received
0928 08:44:00.914358 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:00.914363 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:00.914453 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:00.914547 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:00.915344 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.916560 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.919431 (+ 2871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.920823 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.920924 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:00.921030 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:00.921603 (+ 573us) server_negotiation.cc:299] Negotiation successful
0928 08:44:00.921734 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 08:44:00.922776 2707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:00.902895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34607 (local address 127.0.97.1:33635)
0928 08:44:00.913796 (+ 10901us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:00.913818 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:00.914188 (+ 370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:00.914766 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:00.914770 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:00.914801 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:00.915148 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:00.915170 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.916770 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.916777 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:00.919265 (+ 2488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:00.919279 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.921225 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.921231 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:00.921393 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:00.921431 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 08:44:00.922282 (+ 851us) client_negotiation.cc:240] Negotiation successful
0928 08:44:00.922419 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10750,"thread_start_us":11023,"threads_started":1}
I20260928 08:44:00.923373 2443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "154579534b604b038d04380e3f5b94dc" candidate_uuid: "219436dd00ba4093adc396062d6ef10f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a944f0f60ae43498dd4ef372b97458c" is_pre_election: true
I20260928 08:44:00.923696 2443 raft_consensus.cc:2454] T 154579534b604b038d04380e3f5b94dc P 2a944f0f60ae43498dd4ef372b97458c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 219436dd00ba4093adc396062d6ef10f in term 0.
I20260928 08:44:00.923828 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:00.903717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59361 (local address 127.0.97.3:37393)
0928 08:44:00.903950 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:00.903957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:00.915106 (+ 11149us) server_negotiation.cc:407] Connection header received
0928 08:44:00.915565 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:00.915572 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:00.915650 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:00.915717 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:00.916576 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.917930 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.920651 (+ 2721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.922104 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.922213 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:00.922322 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:00.923318 (+ 996us) server_negotiation.cc:299] Negotiation successful
0928 08:44:00.923472 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 08:44:00.924327 2235 leader_election.cc:302] T 154579534b604b038d04380e3f5b94dc P 219436dd00ba4093adc396062d6ef10f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 219436dd00ba4093adc396062d6ef10f, 2a944f0f60ae43498dd4ef372b97458c; no voters:
I20260928 08:44:00.924749 2658 raft_consensus.cc:2781] T 154579534b604b038d04380e3f5b94dc P 219436dd00ba4093adc396062d6ef10f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:44:00.924852 2658 raft_consensus.cc:482] T 154579534b604b038d04380e3f5b94dc P 219436dd00ba4093adc396062d6ef10f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:44:00.924934 2658 raft_consensus.cc:3037] T 154579534b604b038d04380e3f5b94dc P 219436dd00ba4093adc396062d6ef10f [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:00.926342 2708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:00.903087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37393 (local address 127.0.97.1:59361)
0928 08:44:00.914996 (+ 11909us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:00.915016 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:00.915392 (+ 376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:00.915941 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:00.915946 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:00.915971 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:00.916370 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:00.916381 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.918114 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.918120 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:00.920457 (+ 2337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:00.920472 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.922738 (+ 2266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.922746 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:00.923118 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:00.923153 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 08:44:00.925754 (+ 2601us) client_negotiation.cc:240] Negotiation successful
0928 08:44:00.925921 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11740,"thread_start_us":12037,"threads_started":1}
I20260928 08:44:00.927137 2582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "154579534b604b038d04380e3f5b94dc" candidate_uuid: "219436dd00ba4093adc396062d6ef10f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "469696e9fb2948138ebce9b1bd275d5c" is_pre_election: true
I20260928 08:44:00.927465 2582 raft_consensus.cc:2454] T 154579534b604b038d04380e3f5b94dc P 469696e9fb2948138ebce9b1bd275d5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 219436dd00ba4093adc396062d6ef10f in term 0.
I20260928 08:44:00.928119 2658 raft_consensus.cc:504] T 154579534b604b038d04380e3f5b94dc P 219436dd00ba4093adc396062d6ef10f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:00.928781 2658 leader_election.cc:288] T 154579534b604b038d04380e3f5b94dc P 219436dd00ba4093adc396062d6ef10f [CANDIDATE]: Term 1 election: Requested vote from peers 2a944f0f60ae43498dd4ef372b97458c (127.0.97.2:34607), 469696e9fb2948138ebce9b1bd275d5c (127.0.97.3:37393)
I20260928 08:44:00.929382 2443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "154579534b604b038d04380e3f5b94dc" candidate_uuid: "219436dd00ba4093adc396062d6ef10f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a944f0f60ae43498dd4ef372b97458c"
I20260928 08:44:00.929404 2582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "154579534b604b038d04380e3f5b94dc" candidate_uuid: "219436dd00ba4093adc396062d6ef10f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "469696e9fb2948138ebce9b1bd275d5c"
I20260928 08:44:00.929584 2582 raft_consensus.cc:3037] T 154579534b604b038d04380e3f5b94dc P 469696e9fb2948138ebce9b1bd275d5c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:00.929589 2443 raft_consensus.cc:3037] T 154579534b604b038d04380e3f5b94dc P 2a944f0f60ae43498dd4ef372b97458c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:00.932708 2443 raft_consensus.cc:2454] T 154579534b604b038d04380e3f5b94dc P 2a944f0f60ae43498dd4ef372b97458c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 219436dd00ba4093adc396062d6ef10f in term 1.
I20260928 08:44:00.933074 2582 raft_consensus.cc:2454] T 154579534b604b038d04380e3f5b94dc P 469696e9fb2948138ebce9b1bd275d5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 219436dd00ba4093adc396062d6ef10f in term 1.
I20260928 08:44:00.933080 2235 leader_election.cc:302] T 154579534b604b038d04380e3f5b94dc P 219436dd00ba4093adc396062d6ef10f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 219436dd00ba4093adc396062d6ef10f, 2a944f0f60ae43498dd4ef372b97458c; no voters:
I20260928 08:44:00.933669 2658 raft_consensus.cc:2781] T 154579534b604b038d04380e3f5b94dc P 219436dd00ba4093adc396062d6ef10f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:44:00.934176 2658 raft_consensus.cc:686] T 154579534b604b038d04380e3f5b94dc P 219436dd00ba4093adc396062d6ef10f [term 1 LEADER]: Becoming Leader. State: Replica: 219436dd00ba4093adc396062d6ef10f, State: Running, Role: LEADER
I20260928 08:44:00.934592 2658 consensus_queue.cc:237] T 154579534b604b038d04380e3f5b94dc P 219436dd00ba4093adc396062d6ef10f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:00.938424 2147 catalog_manager.cc:5495] T 154579534b604b038d04380e3f5b94dc P 219436dd00ba4093adc396062d6ef10f reported cstate change: term changed from 0 to 1, leader changed from <none> to 219436dd00ba4093adc396062d6ef10f (127.0.97.1). New cstate: current_term: 1 leader_uuid: "219436dd00ba4093adc396062d6ef10f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } health_report { overall_health: UNKNOWN } } }
I20260928 08:44:00.966286 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:00.959689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41740 (local address 127.0.97.1:45001)
0928 08:44:00.959896 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:00.959903 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:00.959921 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:44:00.960026 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:00.960032 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:00.960125 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:00.960273 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:44:00.960721 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.961800 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.963710 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.964380 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.964493 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:00.964974 (+ 481us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:44:00.965636 (+ 662us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:44:00.965790 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:00.965897 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 08:44:00.965977 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 08:44:00.986860 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:00.977765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48457 (local address 127.0.97.2:34607)
0928 08:44:00.977929 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:00.977935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:00.980433 (+ 2498us) server_negotiation.cc:407] Connection header received
0928 08:44:00.980614 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:00.980621 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:00.980709 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:00.980803 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:44:00.981363 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.982547 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.984670 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.985379 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.985510 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:00.986038 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:44:00.986187 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:44:00.986283 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:00.986438 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 08:44:00.986525 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 08:44:00.986979 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:00.977752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34607 (local address 127.0.97.1:48457)
0928 08:44:00.980313 (+ 2561us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:00.980334 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:00.980481 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:00.980860 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:00.980865 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:00.980879 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:44:00.981196 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:00.981207 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.982684 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.982692 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:00.984518 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:00.984533 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:00.985566 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:00.985572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:00.985774 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:00.985839 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:44:00.986306 (+ 467us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:44:00.986317 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 08:44:00.986402 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 08:44:00.986556 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2385,"thread_start_us":2250,"threads_started":1}
I20260928 08:44:00.997123 2443 raft_consensus.cc:1261] T 154579534b604b038d04380e3f5b94dc P 2a944f0f60ae43498dd4ef372b97458c [term 1 FOLLOWER]: Refusing update from remote peer 219436dd00ba4093adc396062d6ef10f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:44:00.997351 2582 raft_consensus.cc:1261] T 154579534b604b038d04380e3f5b94dc P 469696e9fb2948138ebce9b1bd275d5c [term 1 FOLLOWER]: Refusing update from remote peer 219436dd00ba4093adc396062d6ef10f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:44:00.998064 2709 consensus_queue.cc:1035] T 154579534b604b038d04380e3f5b94dc P 219436dd00ba4093adc396062d6ef10f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:44:01.000774 2709 consensus_queue.cc:1035] T 154579534b604b038d04380e3f5b94dc P 219436dd00ba4093adc396062d6ef10f [LEADER]: Connected to new peer: Peer: permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:44:01.063239 2698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:01.050430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45001 (local address 127.0.97.2:47739)
0928 08:44:01.050763 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:01.050783 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:01.050897 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:01.051655 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:01.051660 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:01.051673 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:44:01.051909 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:01.051921 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:01.054278 (+ 2357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:01.054285 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:01.056027 (+ 1742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:01.056044 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:01.061920 (+ 5876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:01.061927 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:01.062139 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:01.062210 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:44:01.062631 (+ 421us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:44:01.062644 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 08:44:01.062742 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 08:44:01.062899 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191}
I20260928 08:44:01.064011 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:01.051106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47739 (local address 127.0.97.1:45001)
0928 08:44:01.051343 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:01.051349 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:01.051368 (+ 19us) server_negotiation.cc:407] Connection header received
0928 08:44:01.051432 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:01.051438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:01.051520 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:01.051621 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:44:01.052071 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:01.053180 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:01.059905 (+ 6725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:01.061346 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:01.061473 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:01.062390 (+ 917us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:44:01.062517 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:44:01.062606 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:01.063596 (+ 990us) server_negotiation.cc:299] Negotiation successful
0928 08:44:01.063689 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
W20260928 08:44:01.065547 2681 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 08:44:02.673976 2147 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:44:02.674211 2147 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:44:02.678509 2147 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 353e51706cce4c289e9293253cc0d1dd: Sending DeleteTablet for 3 replicas of tablet 154579534b604b038d04380e3f5b94dc
I20260928 08:44:02.679271 2284 tablet_service.cc:1514] Processing DeleteTablet for tablet 154579534b604b038d04380e3f5b94dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41732
I20260928 08:44:02.679363 2424 tablet_service.cc:1514] Processing DeleteTablet for tablet 154579534b604b038d04380e3f5b94dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44852
I20260928 08:44:02.679389 2562 tablet_service.cc:1514] Processing DeleteTablet for tablet 154579534b604b038d04380e3f5b94dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:02 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53086
I20260928 08:44:02.691838 2739 tablet_replica.cc:330] T 154579534b604b038d04380e3f5b94dc P 219436dd00ba4093adc396062d6ef10f: stopping tablet replica
I20260928 08:44:02.691835 2741 tablet_replica.cc:330] T 154579534b604b038d04380e3f5b94dc P 469696e9fb2948138ebce9b1bd275d5c: stopping tablet replica
I20260928 08:44:02.692070 2741 raft_consensus.cc:2229] T 154579534b604b038d04380e3f5b94dc P 469696e9fb2948138ebce9b1bd275d5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:44:02.692226 2739 raft_consensus.cc:2229] T 154579534b604b038d04380e3f5b94dc P 219436dd00ba4093adc396062d6ef10f [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:44:02.692435 2741 raft_consensus.cc:2258] T 154579534b604b038d04380e3f5b94dc P 469696e9fb2948138ebce9b1bd275d5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:02.692847 2739 raft_consensus.cc:2258] T 154579534b604b038d04380e3f5b94dc P 219436dd00ba4093adc396062d6ef10f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:02.693689 2739 ts_tablet_manager.cc:1905] T 154579534b604b038d04380e3f5b94dc P 219436dd00ba4093adc396062d6ef10f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:02.693645 2741 ts_tablet_manager.cc:1905] T 154579534b604b038d04380e3f5b94dc P 469696e9fb2948138ebce9b1bd275d5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:02.696012 2740 tablet_replica.cc:330] T 154579534b604b038d04380e3f5b94dc P 2a944f0f60ae43498dd4ef372b97458c: stopping tablet replica
I20260928 08:44:02.696290 2740 raft_consensus.cc:2229] T 154579534b604b038d04380e3f5b94dc P 2a944f0f60ae43498dd4ef372b97458c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:44:02.696581 2740 raft_consensus.cc:2258] T 154579534b604b038d04380e3f5b94dc P 2a944f0f60ae43498dd4ef372b97458c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:02.697618 2740 ts_tablet_manager.cc:1905] T 154579534b604b038d04380e3f5b94dc P 2a944f0f60ae43498dd4ef372b97458c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:02.700867 2741 ts_tablet_manager.cc:1918] T 154579534b604b038d04380e3f5b94dc P 469696e9fb2948138ebce9b1bd275d5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 08:44:02.700862 2739 ts_tablet_manager.cc:1918] T 154579534b604b038d04380e3f5b94dc P 219436dd00ba4093adc396062d6ef10f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 08:44:02.701004 2739 log.cc:1196] T 154579534b604b038d04380e3f5b94dc P 219436dd00ba4093adc396062d6ef10f: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-0/wal/wals/154579534b604b038d04380e3f5b94dc
I20260928 08:44:02.701040 2741 log.cc:1196] T 154579534b604b038d04380e3f5b94dc P 469696e9fb2948138ebce9b1bd275d5c: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-2/wal/wals/154579534b604b038d04380e3f5b94dc
I20260928 08:44:02.701490 2739 ts_tablet_manager.cc:1939] T 154579534b604b038d04380e3f5b94dc P 219436dd00ba4093adc396062d6ef10f: Deleting consensus metadata
I20260928 08:44:02.701490 2741 ts_tablet_manager.cc:1939] T 154579534b604b038d04380e3f5b94dc P 469696e9fb2948138ebce9b1bd275d5c: Deleting consensus metadata
I20260928 08:44:02.702598 2131 catalog_manager.cc:4841] TS 219436dd00ba4093adc396062d6ef10f (127.0.97.1:45001): tablet 154579534b604b038d04380e3f5b94dc (table test-table [id=1d24c60ce17644709fb3fff93f74e5d1]) successfully deleted
I20260928 08:44:02.702906 2131 catalog_manager.cc:4841] TS 469696e9fb2948138ebce9b1bd275d5c (127.0.97.3:37393): tablet 154579534b604b038d04380e3f5b94dc (table test-table [id=1d24c60ce17644709fb3fff93f74e5d1]) successfully deleted
I20260928 08:44:02.707011 2740 ts_tablet_manager.cc:1918] T 154579534b604b038d04380e3f5b94dc P 2a944f0f60ae43498dd4ef372b97458c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 08:44:02.707121 2740 log.cc:1196] T 154579534b604b038d04380e3f5b94dc P 2a944f0f60ae43498dd4ef372b97458c: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-1/wal/wals/154579534b604b038d04380e3f5b94dc
I20260928 08:44:02.707521 2740 ts_tablet_manager.cc:1939] T 154579534b604b038d04380e3f5b94dc P 2a944f0f60ae43498dd4ef372b97458c: Deleting consensus metadata
I20260928 08:44:02.708524 2132 catalog_manager.cc:4841] TS 2a944f0f60ae43498dd4ef372b97458c (127.0.97.2:34607): tablet 154579534b604b038d04380e3f5b94dc (table test-table [id=1d24c60ce17644709fb3fff93f74e5d1]) successfully deleted
Sep 28 08:44:02 dist-test-slave-n6xs krb5kdc[2102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585042, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 08:44:02 dist-test-slave-n6xs krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585042, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:44:02.738940 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:02.724321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36998 (local address 127.0.97.62:42255)
0928 08:44:02.724881 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:02.724886 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:02.724933 (+ 47us) server_negotiation.cc:407] Connection header received
0928 08:44:02.724986 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:02.724989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:02.725051 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:02.725138 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:02.726390 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:02.727194 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:02.728607 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:02.729000 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:02.729101 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:02.732932 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:02.732937 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:02.732941 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:02.732975 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:02.736502 (+ 3527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:02.737129 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:02.737133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:02.737135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:02.737266 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:02.737589 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:02.737593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:02.737595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:02.737920 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:02.738099 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:02.738380 (+ 281us) server_negotiation.cc:299] Negotiation successful
0928 08:44:02.738551 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":287,"threads_started":1}
I20260928 08:44:02.742354 2147 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36998:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:44:02.742972 2147 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:44:02.750005 2424 tablet_service.cc:1467] Processing CreateTablet for tablet 78052a815e344dfa91925f30e0f2d86f (DEFAULT_TABLE table=test-table [id=c201f68130f74c73a11d11921aacfb07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:02.749985 2284 tablet_service.cc:1467] Processing CreateTablet for tablet 78052a815e344dfa91925f30e0f2d86f (DEFAULT_TABLE table=test-table [id=c201f68130f74c73a11d11921aacfb07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:02.750265 2562 tablet_service.cc:1467] Processing CreateTablet for tablet 78052a815e344dfa91925f30e0f2d86f (DEFAULT_TABLE table=test-table [id=c201f68130f74c73a11d11921aacfb07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:02.750463 2424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78052a815e344dfa91925f30e0f2d86f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:02.750469 2284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78052a815e344dfa91925f30e0f2d86f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:02.750655 2562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78052a815e344dfa91925f30e0f2d86f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:02.756270 2750 tablet_bootstrap.cc:492] T 78052a815e344dfa91925f30e0f2d86f P 219436dd00ba4093adc396062d6ef10f: Bootstrap starting.
I20260928 08:44:02.758502 2750 tablet_bootstrap.cc:654] T 78052a815e344dfa91925f30e0f2d86f P 219436dd00ba4093adc396062d6ef10f: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:02.760334 2752 tablet_bootstrap.cc:492] T 78052a815e344dfa91925f30e0f2d86f P 469696e9fb2948138ebce9b1bd275d5c: Bootstrap starting.
I20260928 08:44:02.760643 2751 tablet_bootstrap.cc:492] T 78052a815e344dfa91925f30e0f2d86f P 2a944f0f60ae43498dd4ef372b97458c: Bootstrap starting.
I20260928 08:44:02.760753 2750 tablet_bootstrap.cc:492] T 78052a815e344dfa91925f30e0f2d86f P 219436dd00ba4093adc396062d6ef10f: No bootstrap required, opened a new log
I20260928 08:44:02.760919 2750 ts_tablet_manager.cc:1397] T 78052a815e344dfa91925f30e0f2d86f P 219436dd00ba4093adc396062d6ef10f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260928 08:44:02.761894 2751 tablet_bootstrap.cc:654] T 78052a815e344dfa91925f30e0f2d86f P 2a944f0f60ae43498dd4ef372b97458c: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:02.761768 2750 raft_consensus.cc:348] T 78052a815e344dfa91925f30e0f2d86f P 219436dd00ba4093adc396062d6ef10f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:02.762068 2752 tablet_bootstrap.cc:654] T 78052a815e344dfa91925f30e0f2d86f P 469696e9fb2948138ebce9b1bd275d5c: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:02.762046 2750 raft_consensus.cc:374] T 78052a815e344dfa91925f30e0f2d86f P 219436dd00ba4093adc396062d6ef10f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:02.762157 2750 raft_consensus.cc:729] T 78052a815e344dfa91925f30e0f2d86f P 219436dd00ba4093adc396062d6ef10f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 219436dd00ba4093adc396062d6ef10f, State: Initialized, Role: FOLLOWER
I20260928 08:44:02.762851 2750 consensus_queue.cc:260] T 78052a815e344dfa91925f30e0f2d86f P 219436dd00ba4093adc396062d6ef10f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:02.764247 2751 tablet_bootstrap.cc:492] T 78052a815e344dfa91925f30e0f2d86f P 2a944f0f60ae43498dd4ef372b97458c: No bootstrap required, opened a new log
I20260928 08:44:02.764433 2751 ts_tablet_manager.cc:1397] T 78052a815e344dfa91925f30e0f2d86f P 2a944f0f60ae43498dd4ef372b97458c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260928 08:44:02.764921 2750 ts_tablet_manager.cc:1428] T 78052a815e344dfa91925f30e0f2d86f P 219436dd00ba4093adc396062d6ef10f: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260928 08:44:02.765368 2751 raft_consensus.cc:348] T 78052a815e344dfa91925f30e0f2d86f P 2a944f0f60ae43498dd4ef372b97458c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:02.765590 2751 raft_consensus.cc:374] T 78052a815e344dfa91925f30e0f2d86f P 2a944f0f60ae43498dd4ef372b97458c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:02.765684 2751 raft_consensus.cc:729] T 78052a815e344dfa91925f30e0f2d86f P 2a944f0f60ae43498dd4ef372b97458c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a944f0f60ae43498dd4ef372b97458c, State: Initialized, Role: FOLLOWER
I20260928 08:44:02.765916 2751 consensus_queue.cc:260] T 78052a815e344dfa91925f30e0f2d86f P 2a944f0f60ae43498dd4ef372b97458c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:02.766520 2751 ts_tablet_manager.cc:1428] T 78052a815e344dfa91925f30e0f2d86f P 2a944f0f60ae43498dd4ef372b97458c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 08:44:02.767486 2752 tablet_bootstrap.cc:492] T 78052a815e344dfa91925f30e0f2d86f P 469696e9fb2948138ebce9b1bd275d5c: No bootstrap required, opened a new log
I20260928 08:44:02.767655 2752 ts_tablet_manager.cc:1397] T 78052a815e344dfa91925f30e0f2d86f P 469696e9fb2948138ebce9b1bd275d5c: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 08:44:02.768505 2752 raft_consensus.cc:348] T 78052a815e344dfa91925f30e0f2d86f P 469696e9fb2948138ebce9b1bd275d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:02.768703 2752 raft_consensus.cc:374] T 78052a815e344dfa91925f30e0f2d86f P 469696e9fb2948138ebce9b1bd275d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:02.768821 2752 raft_consensus.cc:729] T 78052a815e344dfa91925f30e0f2d86f P 469696e9fb2948138ebce9b1bd275d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 469696e9fb2948138ebce9b1bd275d5c, State: Initialized, Role: FOLLOWER
I20260928 08:44:02.769085 2752 consensus_queue.cc:260] T 78052a815e344dfa91925f30e0f2d86f P 469696e9fb2948138ebce9b1bd275d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:02.770033 2752 ts_tablet_manager.cc:1428] T 78052a815e344dfa91925f30e0f2d86f P 469696e9fb2948138ebce9b1bd275d5c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 08:44:02.905602 2730 raft_consensus.cc:482] T 78052a815e344dfa91925f30e0f2d86f P 219436dd00ba4093adc396062d6ef10f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:44:02.905889 2730 raft_consensus.cc:504] T 78052a815e344dfa91925f30e0f2d86f P 219436dd00ba4093adc396062d6ef10f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:02.906617 2730 leader_election.cc:288] T 78052a815e344dfa91925f30e0f2d86f P 219436dd00ba4093adc396062d6ef10f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a944f0f60ae43498dd4ef372b97458c (127.0.97.2:34607), 469696e9fb2948138ebce9b1bd275d5c (127.0.97.3:37393)
I20260928 08:44:02.907186 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78052a815e344dfa91925f30e0f2d86f" candidate_uuid: "219436dd00ba4093adc396062d6ef10f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "469696e9fb2948138ebce9b1bd275d5c" is_pre_election: true
I20260928 08:44:02.907232 2444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78052a815e344dfa91925f30e0f2d86f" candidate_uuid: "219436dd00ba4093adc396062d6ef10f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a944f0f60ae43498dd4ef372b97458c" is_pre_election: true
I20260928 08:44:02.907457 2583 raft_consensus.cc:2454] T 78052a815e344dfa91925f30e0f2d86f P 469696e9fb2948138ebce9b1bd275d5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 219436dd00ba4093adc396062d6ef10f in term 0.
I20260928 08:44:02.907485 2444 raft_consensus.cc:2454] T 78052a815e344dfa91925f30e0f2d86f P 2a944f0f60ae43498dd4ef372b97458c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 219436dd00ba4093adc396062d6ef10f in term 0.
I20260928 08:44:02.907987 2234 leader_election.cc:302] T 78052a815e344dfa91925f30e0f2d86f P 219436dd00ba4093adc396062d6ef10f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 219436dd00ba4093adc396062d6ef10f, 469696e9fb2948138ebce9b1bd275d5c; no voters:
I20260928 08:44:02.908293 2730 raft_consensus.cc:2781] T 78052a815e344dfa91925f30e0f2d86f P 219436dd00ba4093adc396062d6ef10f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:44:02.908412 2730 raft_consensus.cc:482] T 78052a815e344dfa91925f30e0f2d86f P 219436dd00ba4093adc396062d6ef10f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:44:02.908502 2730 raft_consensus.cc:3037] T 78052a815e344dfa91925f30e0f2d86f P 219436dd00ba4093adc396062d6ef10f [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:02.910509 2730 raft_consensus.cc:504] T 78052a815e344dfa91925f30e0f2d86f P 219436dd00ba4093adc396062d6ef10f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:02.911048 2730 leader_election.cc:288] T 78052a815e344dfa91925f30e0f2d86f P 219436dd00ba4093adc396062d6ef10f [CANDIDATE]: Term 1 election: Requested vote from peers 2a944f0f60ae43498dd4ef372b97458c (127.0.97.2:34607), 469696e9fb2948138ebce9b1bd275d5c (127.0.97.3:37393)
I20260928 08:44:02.911521 2444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78052a815e344dfa91925f30e0f2d86f" candidate_uuid: "219436dd00ba4093adc396062d6ef10f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a944f0f60ae43498dd4ef372b97458c"
I20260928 08:44:02.911569 2583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78052a815e344dfa91925f30e0f2d86f" candidate_uuid: "219436dd00ba4093adc396062d6ef10f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "469696e9fb2948138ebce9b1bd275d5c"
I20260928 08:44:02.911736 2444 raft_consensus.cc:3037] T 78052a815e344dfa91925f30e0f2d86f P 2a944f0f60ae43498dd4ef372b97458c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:02.911778 2583 raft_consensus.cc:3037] T 78052a815e344dfa91925f30e0f2d86f P 469696e9fb2948138ebce9b1bd275d5c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:02.914027 2583 raft_consensus.cc:2454] T 78052a815e344dfa91925f30e0f2d86f P 469696e9fb2948138ebce9b1bd275d5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 219436dd00ba4093adc396062d6ef10f in term 1.
I20260928 08:44:02.914302 2444 raft_consensus.cc:2454] T 78052a815e344dfa91925f30e0f2d86f P 2a944f0f60ae43498dd4ef372b97458c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 219436dd00ba4093adc396062d6ef10f in term 1.
I20260928 08:44:02.914444 2234 leader_election.cc:302] T 78052a815e344dfa91925f30e0f2d86f P 219436dd00ba4093adc396062d6ef10f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 219436dd00ba4093adc396062d6ef10f, 469696e9fb2948138ebce9b1bd275d5c; no voters:
I20260928 08:44:02.914765 2730 raft_consensus.cc:2781] T 78052a815e344dfa91925f30e0f2d86f P 219436dd00ba4093adc396062d6ef10f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:44:02.914911 2730 raft_consensus.cc:686] T 78052a815e344dfa91925f30e0f2d86f P 219436dd00ba4093adc396062d6ef10f [term 1 LEADER]: Becoming Leader. State: Replica: 219436dd00ba4093adc396062d6ef10f, State: Running, Role: LEADER
I20260928 08:44:02.915227 2730 consensus_queue.cc:237] T 78052a815e344dfa91925f30e0f2d86f P 219436dd00ba4093adc396062d6ef10f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:02.918045 2145 catalog_manager.cc:5495] T 78052a815e344dfa91925f30e0f2d86f P 219436dd00ba4093adc396062d6ef10f reported cstate change: term changed from 0 to 1, leader changed from <none> to 219436dd00ba4093adc396062d6ef10f (127.0.97.1). New cstate: current_term: 1 leader_uuid: "219436dd00ba4093adc396062d6ef10f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } health_report { overall_health: UNKNOWN } } }
I20260928 08:44:02.963474 2759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:02.958724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41756 (local address 127.0.97.1:45001)
0928 08:44:02.959255 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:02.959259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:02.959278 (+ 19us) server_negotiation.cc:407] Connection header received
0928 08:44:02.959326 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:02.959329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:02.959384 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:02.959463 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:44:02.959892 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:02.960669 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:02.961974 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:02.962380 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:02.962476 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:02.962872 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:44:02.962969 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:44:02.963034 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:02.963173 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 08:44:02.963227 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":311,"threads_started":1}
I20260928 08:44:02.972515 2583 raft_consensus.cc:1261] T 78052a815e344dfa91925f30e0f2d86f P 469696e9fb2948138ebce9b1bd275d5c [term 1 FOLLOWER]: Refusing update from remote peer 219436dd00ba4093adc396062d6ef10f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:44:02.972720 2444 raft_consensus.cc:1261] T 78052a815e344dfa91925f30e0f2d86f P 2a944f0f60ae43498dd4ef372b97458c [term 1 FOLLOWER]: Refusing update from remote peer 219436dd00ba4093adc396062d6ef10f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:44:02.973042 2730 consensus_queue.cc:1035] T 78052a815e344dfa91925f30e0f2d86f P 219436dd00ba4093adc396062d6ef10f [LEADER]: Connected to new peer: Peer: permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:44:02.973310 2709 consensus_queue.cc:1035] T 78052a815e344dfa91925f30e0f2d86f P 219436dd00ba4093adc396062d6ef10f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:44:03.014969 2146 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:44:03.015214 2146 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:44:03.018379 2146 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 353e51706cce4c289e9293253cc0d1dd: Sending DeleteTablet for 3 replicas of tablet 78052a815e344dfa91925f30e0f2d86f
I20260928 08:44:03.019222 2284 tablet_service.cc:1514] Processing DeleteTablet for tablet 78052a815e344dfa91925f30e0f2d86f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:03 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41732
I20260928 08:44:03.019315 2424 tablet_service.cc:1514] Processing DeleteTablet for tablet 78052a815e344dfa91925f30e0f2d86f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:03 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44852
I20260928 08:44:03.019521 2739 tablet_replica.cc:330] T 78052a815e344dfa91925f30e0f2d86f P 219436dd00ba4093adc396062d6ef10f: stopping tablet replica
I20260928 08:44:03.019625 2740 tablet_replica.cc:330] T 78052a815e344dfa91925f30e0f2d86f P 2a944f0f60ae43498dd4ef372b97458c: stopping tablet replica
I20260928 08:44:03.019704 2739 raft_consensus.cc:2229] T 78052a815e344dfa91925f30e0f2d86f P 219436dd00ba4093adc396062d6ef10f [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:44:03.019773 2740 raft_consensus.cc:2229] T 78052a815e344dfa91925f30e0f2d86f P 2a944f0f60ae43498dd4ef372b97458c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:44:03.019991 2740 raft_consensus.cc:2258] T 78052a815e344dfa91925f30e0f2d86f P 2a944f0f60ae43498dd4ef372b97458c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:03.020025 2739 raft_consensus.cc:2258] T 78052a815e344dfa91925f30e0f2d86f P 219436dd00ba4093adc396062d6ef10f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:03.020278 2562 tablet_service.cc:1514] Processing DeleteTablet for tablet 78052a815e344dfa91925f30e0f2d86f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:03 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53086
I20260928 08:44:03.020727 2741 tablet_replica.cc:330] T 78052a815e344dfa91925f30e0f2d86f P 469696e9fb2948138ebce9b1bd275d5c: stopping tablet replica
I20260928 08:44:03.020515 2146 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36998:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 08:44:03.020948 2741 raft_consensus.cc:2229] T 78052a815e344dfa91925f30e0f2d86f P 469696e9fb2948138ebce9b1bd275d5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:44:03.021144 2741 raft_consensus.cc:2258] T 78052a815e344dfa91925f30e0f2d86f P 469696e9fb2948138ebce9b1bd275d5c [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 08:44:03.021220 2146 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:44:03.021277 2740 ts_tablet_manager.cc:1905] T 78052a815e344dfa91925f30e0f2d86f P 2a944f0f60ae43498dd4ef372b97458c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:03.022024 2741 ts_tablet_manager.cc:1905] T 78052a815e344dfa91925f30e0f2d86f P 469696e9fb2948138ebce9b1bd275d5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:03.026207 2739 ts_tablet_manager.cc:1905] T 78052a815e344dfa91925f30e0f2d86f P 219436dd00ba4093adc396062d6ef10f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:03.028328 2741 ts_tablet_manager.cc:1918] T 78052a815e344dfa91925f30e0f2d86f P 469696e9fb2948138ebce9b1bd275d5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:44:03.028486 2741 log.cc:1196] T 78052a815e344dfa91925f30e0f2d86f P 469696e9fb2948138ebce9b1bd275d5c: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-2/wal/wals/78052a815e344dfa91925f30e0f2d86f
I20260928 08:44:03.028666 2424 tablet_service.cc:1467] Processing CreateTablet for tablet b0084dfef11541169c2f85b0e4e063ba (DEFAULT_TABLE table=test-table [id=3ac5eb672d6b415eb6af25774ba67996]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:03.028908 2741 ts_tablet_manager.cc:1939] T 78052a815e344dfa91925f30e0f2d86f P 469696e9fb2948138ebce9b1bd275d5c: Deleting consensus metadata
I20260928 08:44:03.029078 2424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0084dfef11541169c2f85b0e4e063ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:03.029248 2284 tablet_service.cc:1467] Processing CreateTablet for tablet b0084dfef11541169c2f85b0e4e063ba (DEFAULT_TABLE table=test-table [id=3ac5eb672d6b415eb6af25774ba67996]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:03.029610 2284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0084dfef11541169c2f85b0e4e063ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:03.031287 2740 ts_tablet_manager.cc:1918] T 78052a815e344dfa91925f30e0f2d86f P 2a944f0f60ae43498dd4ef372b97458c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:44:03.031430 2740 log.cc:1196] T 78052a815e344dfa91925f30e0f2d86f P 2a944f0f60ae43498dd4ef372b97458c: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-1/wal/wals/78052a815e344dfa91925f30e0f2d86f
I20260928 08:44:03.031841 2740 ts_tablet_manager.cc:1939] T 78052a815e344dfa91925f30e0f2d86f P 2a944f0f60ae43498dd4ef372b97458c: Deleting consensus metadata
I20260928 08:44:03.033020 2132 catalog_manager.cc:4841] TS 2a944f0f60ae43498dd4ef372b97458c (127.0.97.2:34607): tablet 78052a815e344dfa91925f30e0f2d86f (table test-table [id=c201f68130f74c73a11d11921aacfb07]) successfully deleted
I20260928 08:44:03.033887 2562 tablet_service.cc:1467] Processing CreateTablet for tablet b0084dfef11541169c2f85b0e4e063ba (DEFAULT_TABLE table=test-table [id=3ac5eb672d6b415eb6af25774ba67996]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:03.034421 2562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0084dfef11541169c2f85b0e4e063ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:03.035216 2750 tablet_bootstrap.cc:492] T b0084dfef11541169c2f85b0e4e063ba P 219436dd00ba4093adc396062d6ef10f: Bootstrap starting.
I20260928 08:44:03.035715 2751 tablet_bootstrap.cc:492] T b0084dfef11541169c2f85b0e4e063ba P 2a944f0f60ae43498dd4ef372b97458c: Bootstrap starting.
I20260928 08:44:03.036788 2750 tablet_bootstrap.cc:654] T b0084dfef11541169c2f85b0e4e063ba P 219436dd00ba4093adc396062d6ef10f: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:03.037303 2751 tablet_bootstrap.cc:654] T b0084dfef11541169c2f85b0e4e063ba P 2a944f0f60ae43498dd4ef372b97458c: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:03.038898 2750 tablet_bootstrap.cc:492] T b0084dfef11541169c2f85b0e4e063ba P 219436dd00ba4093adc396062d6ef10f: No bootstrap required, opened a new log
I20260928 08:44:03.039073 2750 ts_tablet_manager.cc:1397] T b0084dfef11541169c2f85b0e4e063ba P 219436dd00ba4093adc396062d6ef10f: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 08:44:03.039142 2752 tablet_bootstrap.cc:492] T b0084dfef11541169c2f85b0e4e063ba P 469696e9fb2948138ebce9b1bd275d5c: Bootstrap starting.
I20260928 08:44:03.039752 2131 catalog_manager.cc:4841] TS 469696e9fb2948138ebce9b1bd275d5c (127.0.97.3:37393): tablet 78052a815e344dfa91925f30e0f2d86f (table test-table [id=c201f68130f74c73a11d11921aacfb07]) successfully deleted
I20260928 08:44:03.040076 2750 raft_consensus.cc:348] T b0084dfef11541169c2f85b0e4e063ba P 219436dd00ba4093adc396062d6ef10f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:03.040386 2750 raft_consensus.cc:374] T b0084dfef11541169c2f85b0e4e063ba P 219436dd00ba4093adc396062d6ef10f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:03.040472 2750 raft_consensus.cc:729] T b0084dfef11541169c2f85b0e4e063ba P 219436dd00ba4093adc396062d6ef10f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 219436dd00ba4093adc396062d6ef10f, State: Initialized, Role: FOLLOWER
I20260928 08:44:03.040719 2750 consensus_queue.cc:260] T b0084dfef11541169c2f85b0e4e063ba P 219436dd00ba4093adc396062d6ef10f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:03.041091 2752 tablet_bootstrap.cc:654] T b0084dfef11541169c2f85b0e4e063ba P 469696e9fb2948138ebce9b1bd275d5c: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:03.041350 2750 ts_tablet_manager.cc:1428] T b0084dfef11541169c2f85b0e4e063ba P 219436dd00ba4093adc396062d6ef10f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 08:44:03.041729 2739 ts_tablet_manager.cc:1918] T 78052a815e344dfa91925f30e0f2d86f P 219436dd00ba4093adc396062d6ef10f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:44:03.041936 2739 log.cc:1196] T 78052a815e344dfa91925f30e0f2d86f P 219436dd00ba4093adc396062d6ef10f: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-0/wal/wals/78052a815e344dfa91925f30e0f2d86f
I20260928 08:44:03.042531 2739 ts_tablet_manager.cc:1939] T 78052a815e344dfa91925f30e0f2d86f P 219436dd00ba4093adc396062d6ef10f: Deleting consensus metadata
I20260928 08:44:03.044031 2131 catalog_manager.cc:4841] TS 219436dd00ba4093adc396062d6ef10f (127.0.97.1:45001): tablet 78052a815e344dfa91925f30e0f2d86f (table test-table [id=c201f68130f74c73a11d11921aacfb07]) successfully deleted
I20260928 08:44:03.044778 2751 tablet_bootstrap.cc:492] T b0084dfef11541169c2f85b0e4e063ba P 2a944f0f60ae43498dd4ef372b97458c: No bootstrap required, opened a new log
I20260928 08:44:03.044948 2751 ts_tablet_manager.cc:1397] T b0084dfef11541169c2f85b0e4e063ba P 2a944f0f60ae43498dd4ef372b97458c: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260928 08:44:03.045671 2751 raft_consensus.cc:348] T b0084dfef11541169c2f85b0e4e063ba P 2a944f0f60ae43498dd4ef372b97458c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:03.045894 2751 raft_consensus.cc:374] T b0084dfef11541169c2f85b0e4e063ba P 2a944f0f60ae43498dd4ef372b97458c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:03.045979 2751 raft_consensus.cc:729] T b0084dfef11541169c2f85b0e4e063ba P 2a944f0f60ae43498dd4ef372b97458c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a944f0f60ae43498dd4ef372b97458c, State: Initialized, Role: FOLLOWER
I20260928 08:44:03.046243 2752 tablet_bootstrap.cc:492] T b0084dfef11541169c2f85b0e4e063ba P 469696e9fb2948138ebce9b1bd275d5c: No bootstrap required, opened a new log
I20260928 08:44:03.046214 2751 consensus_queue.cc:260] T b0084dfef11541169c2f85b0e4e063ba P 2a944f0f60ae43498dd4ef372b97458c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:03.046401 2752 ts_tablet_manager.cc:1397] T b0084dfef11541169c2f85b0e4e063ba P 469696e9fb2948138ebce9b1bd275d5c: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260928 08:44:03.046707 2751 ts_tablet_manager.cc:1428] T b0084dfef11541169c2f85b0e4e063ba P 2a944f0f60ae43498dd4ef372b97458c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 08:44:03.047084 2752 raft_consensus.cc:348] T b0084dfef11541169c2f85b0e4e063ba P 469696e9fb2948138ebce9b1bd275d5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:03.047323 2752 raft_consensus.cc:374] T b0084dfef11541169c2f85b0e4e063ba P 469696e9fb2948138ebce9b1bd275d5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:03.047408 2752 raft_consensus.cc:729] T b0084dfef11541169c2f85b0e4e063ba P 469696e9fb2948138ebce9b1bd275d5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 469696e9fb2948138ebce9b1bd275d5c, State: Initialized, Role: FOLLOWER
I20260928 08:44:03.047638 2752 consensus_queue.cc:260] T b0084dfef11541169c2f85b0e4e063ba P 469696e9fb2948138ebce9b1bd275d5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:03.048122 2752 ts_tablet_manager.cc:1428] T b0084dfef11541169c2f85b0e4e063ba P 469696e9fb2948138ebce9b1bd275d5c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 08:44:03.131186 2756 raft_consensus.cc:482] T b0084dfef11541169c2f85b0e4e063ba P 469696e9fb2948138ebce9b1bd275d5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:44:03.131405 2756 raft_consensus.cc:504] T b0084dfef11541169c2f85b0e4e063ba P 469696e9fb2948138ebce9b1bd275d5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:03.132232 2756 leader_election.cc:288] T b0084dfef11541169c2f85b0e4e063ba P 469696e9fb2948138ebce9b1bd275d5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 219436dd00ba4093adc396062d6ef10f (127.0.97.1:45001), 2a944f0f60ae43498dd4ef372b97458c (127.0.97.2:34607)
I20260928 08:44:03.132725 2308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0084dfef11541169c2f85b0e4e063ba" candidate_uuid: "469696e9fb2948138ebce9b1bd275d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "219436dd00ba4093adc396062d6ef10f" is_pre_election: true
I20260928 08:44:03.132776 2444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0084dfef11541169c2f85b0e4e063ba" candidate_uuid: "469696e9fb2948138ebce9b1bd275d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a944f0f60ae43498dd4ef372b97458c" is_pre_election: true
I20260928 08:44:03.132987 2308 raft_consensus.cc:2454] T b0084dfef11541169c2f85b0e4e063ba P 219436dd00ba4093adc396062d6ef10f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 469696e9fb2948138ebce9b1bd275d5c in term 0.
I20260928 08:44:03.133023 2444 raft_consensus.cc:2454] T b0084dfef11541169c2f85b0e4e063ba P 2a944f0f60ae43498dd4ef372b97458c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 469696e9fb2948138ebce9b1bd275d5c in term 0.
I20260928 08:44:03.133461 2515 leader_election.cc:302] T b0084dfef11541169c2f85b0e4e063ba P 469696e9fb2948138ebce9b1bd275d5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a944f0f60ae43498dd4ef372b97458c, 469696e9fb2948138ebce9b1bd275d5c; no voters:
I20260928 08:44:03.133759 2756 raft_consensus.cc:2781] T b0084dfef11541169c2f85b0e4e063ba P 469696e9fb2948138ebce9b1bd275d5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:44:03.133869 2756 raft_consensus.cc:482] T b0084dfef11541169c2f85b0e4e063ba P 469696e9fb2948138ebce9b1bd275d5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:44:03.133936 2756 raft_consensus.cc:3037] T b0084dfef11541169c2f85b0e4e063ba P 469696e9fb2948138ebce9b1bd275d5c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:03.136094 2756 raft_consensus.cc:504] T b0084dfef11541169c2f85b0e4e063ba P 469696e9fb2948138ebce9b1bd275d5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:03.136605 2756 leader_election.cc:288] T b0084dfef11541169c2f85b0e4e063ba P 469696e9fb2948138ebce9b1bd275d5c [CANDIDATE]: Term 1 election: Requested vote from peers 219436dd00ba4093adc396062d6ef10f (127.0.97.1:45001), 2a944f0f60ae43498dd4ef372b97458c (127.0.97.2:34607)
I20260928 08:44:03.137145 2444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0084dfef11541169c2f85b0e4e063ba" candidate_uuid: "469696e9fb2948138ebce9b1bd275d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a944f0f60ae43498dd4ef372b97458c"
I20260928 08:44:03.137145 2308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0084dfef11541169c2f85b0e4e063ba" candidate_uuid: "469696e9fb2948138ebce9b1bd275d5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "219436dd00ba4093adc396062d6ef10f"
I20260928 08:44:03.137353 2308 raft_consensus.cc:3037] T b0084dfef11541169c2f85b0e4e063ba P 219436dd00ba4093adc396062d6ef10f [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:03.137362 2444 raft_consensus.cc:3037] T b0084dfef11541169c2f85b0e4e063ba P 2a944f0f60ae43498dd4ef372b97458c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:03.140287 2444 raft_consensus.cc:2454] T b0084dfef11541169c2f85b0e4e063ba P 2a944f0f60ae43498dd4ef372b97458c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 469696e9fb2948138ebce9b1bd275d5c in term 1.
I20260928 08:44:03.140286 2308 raft_consensus.cc:2454] T b0084dfef11541169c2f85b0e4e063ba P 219436dd00ba4093adc396062d6ef10f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 469696e9fb2948138ebce9b1bd275d5c in term 1.
I20260928 08:44:03.140666 2514 leader_election.cc:302] T b0084dfef11541169c2f85b0e4e063ba P 469696e9fb2948138ebce9b1bd275d5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 219436dd00ba4093adc396062d6ef10f, 469696e9fb2948138ebce9b1bd275d5c; no voters:
I20260928 08:44:03.141024 2756 raft_consensus.cc:2781] T b0084dfef11541169c2f85b0e4e063ba P 469696e9fb2948138ebce9b1bd275d5c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:44:03.141448 2756 raft_consensus.cc:686] T b0084dfef11541169c2f85b0e4e063ba P 469696e9fb2948138ebce9b1bd275d5c [term 1 LEADER]: Becoming Leader. State: Replica: 469696e9fb2948138ebce9b1bd275d5c, State: Running, Role: LEADER
I20260928 08:44:03.141739 2756 consensus_queue.cc:237] T b0084dfef11541169c2f85b0e4e063ba P 469696e9fb2948138ebce9b1bd275d5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } }
I20260928 08:44:03.144795 2146 catalog_manager.cc:5495] T b0084dfef11541169c2f85b0e4e063ba P 469696e9fb2948138ebce9b1bd275d5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 469696e9fb2948138ebce9b1bd275d5c (127.0.97.3). New cstate: current_term: 1 leader_uuid: "469696e9fb2948138ebce9b1bd275d5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "469696e9fb2948138ebce9b1bd275d5c" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37393 } health_report { overall_health: HEALTHY } } }
I20260928 08:44:03.182291 2768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:03.176111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53114 (local address 127.0.97.3:37393)
0928 08:44:03.176745 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:03.176751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:03.176772 (+ 21us) server_negotiation.cc:407] Connection header received
0928 08:44:03.176829 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:03.176834 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:03.176903 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:03.176999 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:44:03.177359 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:03.178544 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:03.179868 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:03.180512 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:03.180644 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:03.181036 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:44:03.181168 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:44:03.181252 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:03.181390 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 08:44:03.181462 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":310,"threads_started":1}
I20260928 08:44:03.193305 2444 raft_consensus.cc:1261] T b0084dfef11541169c2f85b0e4e063ba P 2a944f0f60ae43498dd4ef372b97458c [term 1 FOLLOWER]: Refusing update from remote peer 469696e9fb2948138ebce9b1bd275d5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:44:03.193359 2308 raft_consensus.cc:1261] T b0084dfef11541169c2f85b0e4e063ba P 219436dd00ba4093adc396062d6ef10f [term 1 FOLLOWER]: Refusing update from remote peer 469696e9fb2948138ebce9b1bd275d5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:44:03.193888 2766 consensus_queue.cc:1035] T b0084dfef11541169c2f85b0e4e063ba P 469696e9fb2948138ebce9b1bd275d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a944f0f60ae43498dd4ef372b97458c" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:44:03.194182 2756 consensus_queue.cc:1035] T b0084dfef11541169c2f85b0e4e063ba P 469696e9fb2948138ebce9b1bd275d5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "219436dd00ba4093adc396062d6ef10f" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 08:44:03.214533 2721 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 08:44:04.813616 2146 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:44:04.813846 2146 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36998:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:44:04.818176 2146 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 353e51706cce4c289e9293253cc0d1dd: Sending DeleteTablet for 3 replicas of tablet b0084dfef11541169c2f85b0e4e063ba
I20260928 08:44:04.818967 2284 tablet_service.cc:1514] Processing DeleteTablet for tablet b0084dfef11541169c2f85b0e4e063ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:04 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41732
I20260928 08:44:04.818997 2424 tablet_service.cc:1514] Processing DeleteTablet for tablet b0084dfef11541169c2f85b0e4e063ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:04 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44852
I20260928 08:44:04.819056 2562 tablet_service.cc:1514] Processing DeleteTablet for tablet b0084dfef11541169c2f85b0e4e063ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:04 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53086
I20260928 08:44:04.819747 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 2218
I20260928 08:44:04.829964 2800 tablet_replica.cc:330] T b0084dfef11541169c2f85b0e4e063ba P 2a944f0f60ae43498dd4ef372b97458c: stopping tablet replica
I20260928 08:44:04.830374 2800 raft_consensus.cc:2229] T b0084dfef11541169c2f85b0e4e063ba P 2a944f0f60ae43498dd4ef372b97458c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:44:04.830732 2800 raft_consensus.cc:2258] T b0084dfef11541169c2f85b0e4e063ba P 2a944f0f60ae43498dd4ef372b97458c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:04.831323 2799 tablet_replica.cc:330] T b0084dfef11541169c2f85b0e4e063ba P 469696e9fb2948138ebce9b1bd275d5c: stopping tablet replica
I20260928 08:44:04.831452 2798 tablet_replica.cc:330] T b0084dfef11541169c2f85b0e4e063ba P 219436dd00ba4093adc396062d6ef10f: stopping tablet replica
I20260928 08:44:04.831525 2799 raft_consensus.cc:2229] T b0084dfef11541169c2f85b0e4e063ba P 469696e9fb2948138ebce9b1bd275d5c [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:44:04.831629 2798 raft_consensus.cc:2229] T b0084dfef11541169c2f85b0e4e063ba P 219436dd00ba4093adc396062d6ef10f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:44:04.831890 2798 raft_consensus.cc:2258] T b0084dfef11541169c2f85b0e4e063ba P 219436dd00ba4093adc396062d6ef10f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:04.831909 2799 raft_consensus.cc:2258] T b0084dfef11541169c2f85b0e4e063ba P 469696e9fb2948138ebce9b1bd275d5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:04.832612 2800 ts_tablet_manager.cc:1905] T b0084dfef11541169c2f85b0e4e063ba P 2a944f0f60ae43498dd4ef372b97458c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:04.832830 2798 ts_tablet_manager.cc:1905] T b0084dfef11541169c2f85b0e4e063ba P 219436dd00ba4093adc396062d6ef10f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:04.833151 2799 ts_tablet_manager.cc:1905] T b0084dfef11541169c2f85b0e4e063ba P 469696e9fb2948138ebce9b1bd275d5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:04.839437 2798 ts_tablet_manager.cc:1918] T b0084dfef11541169c2f85b0e4e063ba P 219436dd00ba4093adc396062d6ef10f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 08:44:04.839582 2798 log.cc:1196] T b0084dfef11541169c2f85b0e4e063ba P 219436dd00ba4093adc396062d6ef10f: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-0/wal/wals/b0084dfef11541169c2f85b0e4e063ba
I20260928 08:44:04.839769 2799 ts_tablet_manager.cc:1918] T b0084dfef11541169c2f85b0e4e063ba P 469696e9fb2948138ebce9b1bd275d5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 08:44:04.839911 2799 log.cc:1196] T b0084dfef11541169c2f85b0e4e063ba P 469696e9fb2948138ebce9b1bd275d5c: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-2/wal/wals/b0084dfef11541169c2f85b0e4e063ba
I20260928 08:44:04.840091 2798 ts_tablet_manager.cc:1939] T b0084dfef11541169c2f85b0e4e063ba P 219436dd00ba4093adc396062d6ef10f: Deleting consensus metadata
I20260928 08:44:04.840343 2799 ts_tablet_manager.cc:1939] T b0084dfef11541169c2f85b0e4e063ba P 469696e9fb2948138ebce9b1bd275d5c: Deleting consensus metadata
I20260928 08:44:04.841223 2800 ts_tablet_manager.cc:1918] T b0084dfef11541169c2f85b0e4e063ba P 2a944f0f60ae43498dd4ef372b97458c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 08:44:04.841424 2800 log.cc:1196] T b0084dfef11541169c2f85b0e4e063ba P 2a944f0f60ae43498dd4ef372b97458c: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790585022442112-388-0/minicluster-data/ts-1/wal/wals/b0084dfef11541169c2f85b0e4e063ba
I20260928 08:44:04.841413 2131 catalog_manager.cc:4841] TS 469696e9fb2948138ebce9b1bd275d5c (127.0.97.3:37393): tablet b0084dfef11541169c2f85b0e4e063ba (table test-table [id=3ac5eb672d6b415eb6af25774ba67996]) successfully deleted
I20260928 08:44:04.842016 2800 ts_tablet_manager.cc:1939] T b0084dfef11541169c2f85b0e4e063ba P 2a944f0f60ae43498dd4ef372b97458c: Deleting consensus metadata
I20260928 08:44:04.842258 2131 catalog_manager.cc:4841] TS 219436dd00ba4093adc396062d6ef10f (127.0.97.1:45001): tablet b0084dfef11541169c2f85b0e4e063ba (table test-table [id=3ac5eb672d6b415eb6af25774ba67996]) successfully deleted
I20260928 08:44:04.842952 2132 catalog_manager.cc:4841] TS 2a944f0f60ae43498dd4ef372b97458c (127.0.97.2:34607): tablet b0084dfef11541169c2f85b0e4e063ba (table test-table [id=3ac5eb672d6b415eb6af25774ba67996]) successfully deleted
Sep 28 08:44:04 dist-test-slave-n6xs krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585042, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:44:04.856245 2802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:04.822969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35302 (local address 127.0.97.1:45001)
0928 08:44:04.842060 (+ 19091us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:04.842064 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:04.842106 (+ 42us) server_negotiation.cc:407] Connection header received
0928 08:44:04.842164 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:04.842170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:04.842256 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:04.842372 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:04.843773 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:04.845120 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:04.846386 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:04.846810 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:04.846943 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:04.850834 (+ 3891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:04.850848 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:04.850866 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:04.850894 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:04.853557 (+ 2663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:04.854202 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:04.854210 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:04.854212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:04.854348 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:04.854715 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:04.854722 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:04.854725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:04.855026 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:04.855350 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:04.855640 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 08:44:04.855882 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18988,"thread_start_us":18914,"threads_started":1}
W20260928 08:44:04.856886 2349 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35302
W20260928 08:44:04.857452 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 2218: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:44:04.857610 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 2218
I20260928 08:44:04.880338 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 2358
Sep 28 08:44:04 dist-test-slave-n6xs krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585042, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 08:44:04.897997 2803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:04.881053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56862 (local address 127.0.97.2:34607)
0928 08:44:04.881502 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:04.881507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:04.881522 (+ 15us) server_negotiation.cc:407] Connection header received
0928 08:44:04.881581 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:04.881585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:04.881649 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:04.881737 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:04.882946 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:04.883728 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:04.884987 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:04.885432 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:04.885549 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:04.891551 (+ 6002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:04.891570 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:04.891590 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:04.891634 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:04.895437 (+ 3803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:04.896156 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:04.896163 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:04.896165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:04.896284 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:04.896719 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:04.896722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:04.896724 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:04.897050 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:04.897254 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:04.897555 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 08:44:04.897734 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":230,"threads_started":1}
W20260928 08:44:04.898608 2489 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56862
W20260928 08:44:04.899227 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 2358: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:44:04.899408 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 2358
I20260928 08:44:04.923795 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 2498
Sep 28 08:44:04 dist-test-slave-n6xs krb5kdc[2102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585042, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 08:44:04.939016 2804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:04.924537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39180 (local address 127.0.97.3:37393)
0928 08:44:04.925179 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:04.925184 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:04.925201 (+ 17us) server_negotiation.cc:407] Connection header received
0928 08:44:04.925254 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:04.925258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:04.925324 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:04.925420 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:04.926832 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:04.927631 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:04.928884 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:04.929296 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:04.929398 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:04.933676 (+ 4278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:04.933692 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:04.933709 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:04.933741 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:04.936681 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:04.937288 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:04.937295 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:04.937297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:04.937406 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:04.937719 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:04.937722 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:04.937723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:04.938029 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:04.938234 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:04.938484 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 08:44:04.938636 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":297,"threads_started":1}
W20260928 08:44:04.939960 2628 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39180
W20260928 08:44:04.940692 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 2498: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:44:04.940851 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 2498
I20260928 08:44:04.964741 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 2115
I20260928 08:44:04.966176 2187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:05.080039 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 2115
2026-09-28T08:44:05Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8311 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:44:05 dist-test-slave-n6xs krb5kdc[2808](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:44:05 dist-test-slave-n6xs krb5kdc[2808](info): set up 2 sockets
Sep 28 08:44:05 dist-test-slave-n6xs krb5kdc[2808](info): commencing operation
krb5kdc: starting...
W20260928 08:44:06.168215 388 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:44:06 dist-test-slave-n6xs krb5kdc[2808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:44:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:44:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:44:06.355911 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40075
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40075
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:06.675243 2821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:06.675529 2821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:06.675666 2821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:06.682899 2821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:44:06.683003 2821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:06.683101 2821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:06.683205 2821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:44:06.683270 2821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:44:06.693046 2821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:06.694337 2821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:06.695634 2821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:06.707530 2826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:06.707568 2829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:06.707444 2827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:06.708359 2821 server_base.cc:1034] running on GCE node
I20260928 08:44:06.709091 2821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:06.710444 2821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:06.711685 2821 hybrid_clock.cc:648] HybridClock initialized: now 1790585046711654 us; error 57 us; skew 500 ppm
I20260928 08:44:06.712038 2821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:06 dist-test-slave-n6xs krb5kdc[2808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585046, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:06.714974 2821 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:44:06.716502 2821 webserver.cc:460] Webserver started at http://127.0.97.62:35373/ using document root <none> and password file <none>
I20260928 08:44:06.717185 2821 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:06.717327 2821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:06.717636 2821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:06.720538 2821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "9a901fecaaf34e80a0607a1dfea8d4b4"
format_stamp: "Formatted at 2026-09-28 08:44:06 on dist-test-slave-n6xs"
server_key: "dd51c207c7d11289b5d473227158def5"
server_key_iv: "fa42c5607097ec175ce7ce6ecdf5d475"
server_key_version: "encryptionkey@0"
I20260928 08:44:06.721294 2821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "9a901fecaaf34e80a0607a1dfea8d4b4"
format_stamp: "Formatted at 2026-09-28 08:44:06 on dist-test-slave-n6xs"
server_key: "dd51c207c7d11289b5d473227158def5"
server_key_iv: "fa42c5607097ec175ce7ce6ecdf5d475"
server_key_version: "encryptionkey@0"
I20260928 08:44:06.726126 2821 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 08:44:06.729460 2836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:06.730751 2821 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:44:06.730933 2821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "9a901fecaaf34e80a0607a1dfea8d4b4"
format_stamp: "Formatted at 2026-09-28 08:44:06 on dist-test-slave-n6xs"
server_key: "dd51c207c7d11289b5d473227158def5"
server_key_iv: "fa42c5607097ec175ce7ce6ecdf5d475"
server_key_version: "encryptionkey@0"
I20260928 08:44:06.731132 2821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:06.754953 2821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:06.758948 2821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:06.759238 2821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:06.778376 2821 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40075
I20260928 08:44:06.778393 2887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40075 every 8 connection(s)
I20260928 08:44:06.779874 2821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:44:06.782802 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 2821
I20260928 08:44:06.782999 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:44:06.783769 388 external_mini_cluster.cc:1428] Setting key f77be82dedfb38a39ffe59085b72f4df
I20260928 08:44:06.784281 2888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:06.798748 2888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a901fecaaf34e80a0607a1dfea8d4b4: Bootstrap starting.
Sep 28 08:44:06 dist-test-slave-n6xs krb5kdc[2808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:44:06.801884 2888 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a901fecaaf34e80a0607a1dfea8d4b4: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:06.802954 2888 log.cc:824] T 00000000000000000000000000000000 P 9a901fecaaf34e80a0607a1dfea8d4b4: Log is configured to *not* fsync() on all Append() calls
I20260928 08:44:06.805763 2888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a901fecaaf34e80a0607a1dfea8d4b4: No bootstrap required, opened a new log
I20260928 08:44:06.807472 2891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:06.785903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52700 (local address 127.0.97.62:40075)
0928 08:44:06.786736 (+ 833us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:06.786752 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:06.786794 (+ 42us) server_negotiation.cc:407] Connection header received
0928 08:44:06.788909 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:06.788932 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:06.789341 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:06.789698 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:06.791548 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:06.792953 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:06.794777 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:06.795470 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:06.795628 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:06.800371 (+ 4743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:06.800379 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:06.800393 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:06.800429 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:06.803636 (+ 3207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:06.804351 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:06.804361 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:06.804376 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:06.804530 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:06.804914 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:06.804921 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:06.804924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:06.805406 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:06.805709 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:06.806112 (+ 403us) server_negotiation.cc:299] Negotiation successful
0928 08:44:06.806523 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":417,"threads_started":1}
I20260928 08:44:06.813335 2888 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a901fecaaf34e80a0607a1dfea8d4b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a901fecaaf34e80a0607a1dfea8d4b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40075 } }
I20260928 08:44:06.813731 2888 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a901fecaaf34e80a0607a1dfea8d4b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:06.813845 2888 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a901fecaaf34e80a0607a1dfea8d4b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a901fecaaf34e80a0607a1dfea8d4b4, State: Initialized, Role: FOLLOWER
I20260928 08:44:06.814397 2888 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a901fecaaf34e80a0607a1dfea8d4b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a901fecaaf34e80a0607a1dfea8d4b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40075 } }
I20260928 08:44:06.814675 2888 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a901fecaaf34e80a0607a1dfea8d4b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:44:06.814805 2888 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a901fecaaf34e80a0607a1dfea8d4b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:44:06.815034 2888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a901fecaaf34e80a0607a1dfea8d4b4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:06.817286 2888 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a901fecaaf34e80a0607a1dfea8d4b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a901fecaaf34e80a0607a1dfea8d4b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40075 } }
I20260928 08:44:06.817801 2888 leader_election.cc:302] T 00000000000000000000000000000000 P 9a901fecaaf34e80a0607a1dfea8d4b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a901fecaaf34e80a0607a1dfea8d4b4; no voters:
I20260928 08:44:06.818356 2888 leader_election.cc:288] T 00000000000000000000000000000000 P 9a901fecaaf34e80a0607a1dfea8d4b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:44:06.818491 2893 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a901fecaaf34e80a0607a1dfea8d4b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:44:06.819598 2893 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a901fecaaf34e80a0607a1dfea8d4b4 [term 1 LEADER]: Becoming Leader. State: Replica: 9a901fecaaf34e80a0607a1dfea8d4b4, State: Running, Role: LEADER
I20260928 08:44:06.820154 2893 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a901fecaaf34e80a0607a1dfea8d4b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a901fecaaf34e80a0607a1dfea8d4b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40075 } }
I20260928 08:44:06.820545 2888 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a901fecaaf34e80a0607a1dfea8d4b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:44:06.823032 2895 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a901fecaaf34e80a0607a1dfea8d4b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a901fecaaf34e80a0607a1dfea8d4b4. Latest consensus state: current_term: 1 leader_uuid: "9a901fecaaf34e80a0607a1dfea8d4b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a901fecaaf34e80a0607a1dfea8d4b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40075 } } }
I20260928 08:44:06.823477 2895 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a901fecaaf34e80a0607a1dfea8d4b4 [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:06.823593 2894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a901fecaaf34e80a0607a1dfea8d4b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a901fecaaf34e80a0607a1dfea8d4b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a901fecaaf34e80a0607a1dfea8d4b4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40075 } } }
I20260928 08:44:06.824004 2894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a901fecaaf34e80a0607a1dfea8d4b4 [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:06.828943 2902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:44:06.833024 2902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:44:06.841908 2902 catalog_manager.cc:1347] Generated new cluster ID: 7c0a0bd710344bc28d91448db5df9281
I20260928 08:44:06.842011 2902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:44:06.902043 2902 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:44:06.903450 2902 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:44:06.911260 2902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a901fecaaf34e80a0607a1dfea8d4b4: Generated new TSK 0
I20260928 08:44:06.912135 2902 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:44:06.960668 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40075
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:07.322548 2914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:07.322813 2914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:07.322903 2914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:07.330085 2914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:07.330190 2914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:07.330302 2914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:44:07.339926 2914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:07.341183 2914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:07.342420 2914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:07.355034 2922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:07.355104 2919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:07.355113 2920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:07.355728 2914 server_base.cc:1034] running on GCE node
I20260928 08:44:07.356323 2914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:07.357197 2914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:07.358392 2914 hybrid_clock.cc:648] HybridClock initialized: now 1790585047358332 us; error 53 us; skew 500 ppm
I20260928 08:44:07.358681 2914 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:07 dist-test-slave-n6xs krb5kdc[2808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:07.361987 2914 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:44:07.363415 2914 webserver.cc:460] Webserver started at http://127.0.97.1:39915/ using document root <none> and password file <none>
I20260928 08:44:07.364140 2914 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:07.364257 2914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:07.364559 2914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:07.367300 2914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-0/data/instance:
uuid: "d791ed3ae149444cac29d37b36499ce5"
format_stamp: "Formatted at 2026-09-28 08:44:07 on dist-test-slave-n6xs"
server_key: "a0c6212e5abdd489313eec6038a9e3bb"
server_key_iv: "4cb5c6c0a69c0057b1e889132ce1cd59"
server_key_version: "encryptionkey@0"
I20260928 08:44:07.368120 2914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-0/wal/instance:
uuid: "d791ed3ae149444cac29d37b36499ce5"
format_stamp: "Formatted at 2026-09-28 08:44:07 on dist-test-slave-n6xs"
server_key: "a0c6212e5abdd489313eec6038a9e3bb"
server_key_iv: "4cb5c6c0a69c0057b1e889132ce1cd59"
server_key_version: "encryptionkey@0"
I20260928 08:44:07.373152 2914 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 08:44:07.376422 2929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:07.377656 2914 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:44:07.377815 2914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-0/wal
uuid: "d791ed3ae149444cac29d37b36499ce5"
format_stamp: "Formatted at 2026-09-28 08:44:07 on dist-test-slave-n6xs"
server_key: "a0c6212e5abdd489313eec6038a9e3bb"
server_key_iv: "4cb5c6c0a69c0057b1e889132ce1cd59"
server_key_version: "encryptionkey@0"
I20260928 08:44:07.378002 2914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:07.399775 2914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:07.404114 2914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:07.404390 2914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:07.405262 2914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:44:07.406463 2914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:44:07.406562 2914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:07.406706 2914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:44:07.406775 2914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:07.447024 2914 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:32835
I20260928 08:44:07.447108 3041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:32835 every 8 connection(s)
I20260928 08:44:07.448336 2914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:44:07.452711 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 2914
I20260928 08:44:07.452901 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:44:07.453474 388 external_mini_cluster.cc:1428] Setting key 8aec0b047097fea31b14c64a1283c991
Sep 28 08:44:07 dist-test-slave-n6xs krb5kdc[2808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:44:07.469739 3044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:07.450879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50821 (local address 127.0.97.62:40075)
0928 08:44:07.451335 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:07.451339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:07.452352 (+ 1013us) server_negotiation.cc:407] Connection header received
0928 08:44:07.454255 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:07.454261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:07.454339 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:07.454459 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:07.456272 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:07.457542 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:07.459130 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:07.459879 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:07.460020 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:07.464025 (+ 4005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:07.464030 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:07.464033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:07.464077 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:07.466677 (+ 2600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:07.467433 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:07.467438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:07.467442 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:07.467570 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:07.468072 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:07.468077 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:07.468081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:07.468332 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:07.468482 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:07.469167 (+ 685us) server_negotiation.cc:299] Negotiation successful
0928 08:44:07.469388 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":220,"threads_started":1}
I20260928 08:44:07.470692 3045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:07.451450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40075 (local address 127.0.97.1:50821)
0928 08:44:07.452207 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:07.452243 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:07.453966 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:07.454740 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:07.454750 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:07.455454 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:07.456102 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:07.456120 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:07.457693 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:07.457697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:07.458991 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:07.459001 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:07.460074 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:07.460079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:07.460255 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:07.461001 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:07.461034 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:07.463756 (+ 2722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:07.466822 (+ 3066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:07.466830 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:07.466845 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:07.467281 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:07.467700 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:07.467707 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:07.467711 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:07.467953 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:07.468466 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:07.468474 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:07.468812 (+ 338us) client_negotiation.cc:769] Sending connection context
0928 08:44:07.469066 (+ 254us) client_negotiation.cc:240] Negotiation successful
0928 08:44:07.469552 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":355,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 08:44:07.472497 3042 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40075
I20260928 08:44:07.472817 3042 heartbeater.cc:464] Registering TS with master...
I20260928 08:44:07.473598 3042 heartbeater.cc:511] Master 127.0.97.62:40075 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260928 08:44:07.475929 2853 ts_manager.cc:194] Registered new tserver with Master: d791ed3ae149444cac29d37b36499ce5 (127.0.97.1:32835)
I20260928 08:44:07.477396 2853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 08:44:07.497371 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40075
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:07.819234 3048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:07.819578 3048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:07.819767 3048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:07.827275 3048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:07.827423 3048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:07.827643 3048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:44:07.837594 3048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:07.838903 3048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:07.840258 3048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:07.853662 3054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:07.853762 3053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:07.853734 3056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:07.854728 3048 server_base.cc:1034] running on GCE node
I20260928 08:44:07.855332 3048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:07.856384 3048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:07.857640 3048 hybrid_clock.cc:648] HybridClock initialized: now 1790585047857602 us; error 56 us; skew 500 ppm
I20260928 08:44:07.857990 3048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:07 dist-test-slave-n6xs krb5kdc[2808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:07.861328 3048 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:44:07.862888 3048 webserver.cc:460] Webserver started at http://127.0.97.2:36365/ using document root <none> and password file <none>
I20260928 08:44:07.863655 3048 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:07.863852 3048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:07.864209 3048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:07.867184 3048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-1/data/instance:
uuid: "6fce9bff9bf5432290dcf358af32d4cb"
format_stamp: "Formatted at 2026-09-28 08:44:07 on dist-test-slave-n6xs"
server_key: "bc7c3f4ec2fe56a1e64091d271d6b52e"
server_key_iv: "0e2658676103423e13fa0a6eaa7b6bf1"
server_key_version: "encryptionkey@0"
I20260928 08:44:07.868126 3048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-1/wal/instance:
uuid: "6fce9bff9bf5432290dcf358af32d4cb"
format_stamp: "Formatted at 2026-09-28 08:44:07 on dist-test-slave-n6xs"
server_key: "bc7c3f4ec2fe56a1e64091d271d6b52e"
server_key_iv: "0e2658676103423e13fa0a6eaa7b6bf1"
server_key_version: "encryptionkey@0"
I20260928 08:44:07.873431 3048 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 08:44:07.877038 3063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:07.878367 3048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:44:07.878626 3048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-1/wal
uuid: "6fce9bff9bf5432290dcf358af32d4cb"
format_stamp: "Formatted at 2026-09-28 08:44:07 on dist-test-slave-n6xs"
server_key: "bc7c3f4ec2fe56a1e64091d271d6b52e"
server_key_iv: "0e2658676103423e13fa0a6eaa7b6bf1"
server_key_version: "encryptionkey@0"
I20260928 08:44:07.878845 3048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:07.907574 3048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:07.912292 3048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:07.912607 3048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:07.913563 3048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:44:07.914943 3048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:44:07.915052 3048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:07.915207 3048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:44:07.915283 3048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:07.950142 3048 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:43031
I20260928 08:44:07.950237 3175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:43031 every 8 connection(s)
I20260928 08:44:07.951514 3048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:44:07.955428 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 3048
I20260928 08:44:07.955602 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:44:07.956154 388 external_mini_cluster.cc:1428] Setting key 96561564e8d47c8bcc6abbf85bfc9f04
Sep 28 08:44:07 dist-test-slave-n6xs krb5kdc[2808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:44:07.974963 3044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:07.954043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52103 (local address 127.0.97.62:40075)
0928 08:44:07.954237 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:07.954241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:07.955457 (+ 1216us) server_negotiation.cc:407] Connection header received
0928 08:44:07.957671 (+ 2214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:07.957676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:07.957782 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:07.957893 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:07.959912 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:07.960623 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:07.963125 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:07.963601 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:07.963714 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:07.967897 (+ 4183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:07.967906 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:07.967914 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:07.967957 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:07.971543 (+ 3586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:07.972556 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:07.972560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:07.972563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:07.972705 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:07.973212 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:07.973216 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:07.973218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:07.973465 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:07.973626 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:07.974543 (+ 917us) server_negotiation.cc:299] Negotiation successful
0928 08:44:07.974720 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260928 08:44:07.976488 3178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:07.954556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40075 (local address 127.0.97.2:52103)
0928 08:44:07.955289 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:07.955343 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:07.957408 (+ 2065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:07.958121 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:07.958154 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:07.958847 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:07.959661 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:07.959689 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:07.960780 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:07.960787 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:07.962957 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:07.962973 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:07.963855 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:07.963874 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:07.964239 (+ 365us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:07.964982 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:07.965020 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:07.967645 (+ 2625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:07.971714 (+ 4069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:07.971725 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:07.971745 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:07.972386 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:07.972824 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:07.972832 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:07.972837 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:07.973088 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:07.973645 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:07.973653 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:07.974095 (+ 442us) client_negotiation.cc:769] Sending connection context
0928 08:44:07.974406 (+ 311us) client_negotiation.cc:240] Negotiation successful
0928 08:44:07.974976 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":337,"threads_started":1}
I20260928 08:44:07.977844 3176 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40075
I20260928 08:44:07.978163 3176 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260928 08:44:07.978902 3176 heartbeater.cc:511] Master 127.0.97.62:40075 requested a full tablet report, sending...
I20260928 08:44:07.980630 2853 ts_manager.cc:194] Registered new tserver with Master: 6fce9bff9bf5432290dcf358af32d4cb (127.0.97.2:43031)
I20260928 08:44:07.981539 2853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:52103
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:44:08.001808 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40075
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:08.315151 3181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:08.315423 3181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:08.315616 3181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:08.323177 3181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:08.323287 3181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:08.323448 3181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:44:08.333251 3181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:08.334558 3181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:08.335937 3181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:08.348862 3186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:08.348932 3189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:08.348868 3187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:08.349809 3181 server_base.cc:1034] running on GCE node
I20260928 08:44:08.350440 3181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:08.351445 3181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:08.352691 3181 hybrid_clock.cc:648] HybridClock initialized: now 1790585048352634 us; error 50 us; skew 500 ppm
I20260928 08:44:08.353034 3181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:08 dist-test-slave-n6xs krb5kdc[2808](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585048, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:08.356459 3181 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:44:08.357918 3181 webserver.cc:460] Webserver started at http://127.0.97.3:35747/ using document root <none> and password file <none>
I20260928 08:44:08.358613 3181 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:08.358793 3181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:08.359109 3181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:08.362258 3181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-2/data/instance:
uuid: "f7604cbdd54f42c1806742bb10a5cd04"
format_stamp: "Formatted at 2026-09-28 08:44:08 on dist-test-slave-n6xs"
server_key: "248ad9a1f60508f1d0683be3b08ab579"
server_key_iv: "205160d3eedff749898678967b791058"
server_key_version: "encryptionkey@0"
I20260928 08:44:08.363384 3181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-2/wal/instance:
uuid: "f7604cbdd54f42c1806742bb10a5cd04"
format_stamp: "Formatted at 2026-09-28 08:44:08 on dist-test-slave-n6xs"
server_key: "248ad9a1f60508f1d0683be3b08ab579"
server_key_iv: "205160d3eedff749898678967b791058"
server_key_version: "encryptionkey@0"
I20260928 08:44:08.368762 3181 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 08:44:08.372264 3196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:08.373626 3181 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 08:44:08.373803 3181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-2/wal
uuid: "f7604cbdd54f42c1806742bb10a5cd04"
format_stamp: "Formatted at 2026-09-28 08:44:08 on dist-test-slave-n6xs"
server_key: "248ad9a1f60508f1d0683be3b08ab579"
server_key_iv: "205160d3eedff749898678967b791058"
server_key_version: "encryptionkey@0"
I20260928 08:44:08.374011 3181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:08.386202 3181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:08.390439 3181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:08.390720 3181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:08.391630 3181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:44:08.393025 3181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:44:08.393128 3181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:08.393271 3181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:44:08.393344 3181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:08.427129 3181 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33879
I20260928 08:44:08.427168 3308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33879 every 8 connection(s)
I20260928 08:44:08.428428 3181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:44:08.438906 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 3181
I20260928 08:44:08.439077 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:44:08.439620 388 external_mini_cluster.cc:1428] Setting key 0ea0f38bdc2f22dbfa4211c99aa09f53
Sep 28 08:44:08 dist-test-slave-n6xs krb5kdc[2808](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585048, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:44:08.448812 3044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:08.430887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:37043 (local address 127.0.97.62:40075)
0928 08:44:08.431062 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:08.431066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:08.432246 (+ 1180us) server_negotiation.cc:407] Connection header received
0928 08:44:08.434036 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:08.434043 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:08.434116 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:08.434209 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:08.435764 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:08.436600 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:08.438110 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:08.438486 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:08.438591 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:08.442733 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:08.442742 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:08.442747 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:08.442792 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:08.445524 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:08.446192 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:08.446196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:08.446198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:08.446340 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:08.446817 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:08.446821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:08.446823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:08.447089 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:08.447216 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:08.448187 (+ 971us) server_negotiation.cc:299] Negotiation successful
0928 08:44:08.448369 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 08:44:08.449817 3311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:08.431374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40075 (local address 127.0.97.3:37043)
0928 08:44:08.432076 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:08.432120 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:08.433711 (+ 1591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:08.434450 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:08.434459 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:08.434960 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:08.435604 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:08.435620 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:08.436739 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:08.436742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:08.437941 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:08.437952 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:08.438734 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:08.438741 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:08.439029 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:08.439795 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:08.439850 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:08.442472 (+ 2622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:08.445651 (+ 3179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:08.445657 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:08.445670 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:08.446088 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:08.446454 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:08.446458 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:08.446461 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:08.446695 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:08.447221 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:08.447229 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:08.447679 (+ 450us) client_negotiation.cc:769] Sending connection context
0928 08:44:08.448036 (+ 357us) client_negotiation.cc:240] Negotiation successful
0928 08:44:08.448533 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":276,"threads_started":1}
I20260928 08:44:08.451181 3309 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40075
I20260928 08:44:08.451520 3309 heartbeater.cc:464] Registering TS with master...
I20260928 08:44:08.452273 3309 heartbeater.cc:511] Master 127.0.97.62:40075 requested a full tablet report, sending...
I20260928 08:44:08.453761 2853 ts_manager.cc:194] Registered new tserver with Master: f7604cbdd54f42c1806742bb10a5cd04 (127.0.97.3:33879)
I20260928 08:44:08.454741 2853 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:37043
I20260928 08:44:08.455076 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:44:08.481142 3042 heartbeater.cc:503] Master 127.0.97.62:40075 was elected leader, sending a full tablet report...
W20260928 08:44:08.485092 3044 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:44:08.482477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52704 (local address 127.0.97.62:40075)
0928 08:44:08.482667 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:08.482675 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:08.483157 (+ 482us) server_negotiation.cc:407] Connection header received
0928 08:44:08.483360 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:08.483368 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:08.483440 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:08.483539 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:08.484815 (+ 1276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52704: BlockingRecv error: recv got EOF from 127.0.0.1:52704 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 08:44:08.488027 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 2914
W20260928 08:44:08.490661 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 2914: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:32835: server requires authentication, but client does not have Kerberos credentials available
I20260928 08:44:08.490846 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 2914
I20260928 08:44:08.507045 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 3048
W20260928 08:44:08.509608 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 3048: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:43031: server requires authentication, but client does not have Kerberos credentials available
I20260928 08:44:08.509747 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 3048
I20260928 08:44:08.525079 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 3181
W20260928 08:44:08.527773 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 3181: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:33879: server requires authentication, but client does not have Kerberos credentials available
I20260928 08:44:08.527992 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 3181
I20260928 08:44:08.543346 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 2821
I20260928 08:44:08.544539 2883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:08.645061 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 2821
2026-09-28T08:44:08Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3563 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:44:08 dist-test-slave-n6xs krb5kdc[3326](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:44:08 dist-test-slave-n6xs krb5kdc[3326](info): set up 2 sockets
Sep 28 08:44:08 dist-test-slave-n6xs krb5kdc[3326](info): commencing operation
krb5kdc: starting...
W20260928 08:44:09.732203 388 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:44:09 dist-test-slave-n6xs krb5kdc[3326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:44:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:44:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:44:09.920885 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38327
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:10.240324 3339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:10.240636 3339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:10.240803 3339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:10.248193 3339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:44:10.248301 3339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:10.248389 3339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:10.248472 3339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:44:10.248584 3339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:44:10.258796 3339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:10.260181 3339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:10.261564 3339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:10.273653 3344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:10.273692 3347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:10.273720 3345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:10.274520 3339 server_base.cc:1034] running on GCE node
I20260928 08:44:10.275426 3339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:10.277037 3339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:10.278283 3339 hybrid_clock.cc:648] HybridClock initialized: now 1790585050278218 us; error 55 us; skew 500 ppm
I20260928 08:44:10.278712 3339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:10 dist-test-slave-n6xs krb5kdc[3326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585050, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:10.286324 3339 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:44:10.288118 3339 webserver.cc:460] Webserver started at http://127.0.97.62:37423/ using document root <none> and password file <none>
I20260928 08:44:10.288829 3339 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:10.288959 3339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:10.289283 3339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:10.292337 3339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "666de050916c4e4787b0b5130a8afeb1"
format_stamp: "Formatted at 2026-09-28 08:44:10 on dist-test-slave-n6xs"
server_key: "25402f4e6c15d2214a549508445c6a84"
server_key_iv: "c0c2c011c3fbbb86910eaab09f1c38b1"
server_key_version: "encryptionkey@0"
I20260928 08:44:10.293134 3339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "666de050916c4e4787b0b5130a8afeb1"
format_stamp: "Formatted at 2026-09-28 08:44:10 on dist-test-slave-n6xs"
server_key: "25402f4e6c15d2214a549508445c6a84"
server_key_iv: "c0c2c011c3fbbb86910eaab09f1c38b1"
server_key_version: "encryptionkey@0"
I20260928 08:44:10.298291 3339 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:44:10.301807 3354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:10.303262 3339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:44:10.303452 3339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "666de050916c4e4787b0b5130a8afeb1"
format_stamp: "Formatted at 2026-09-28 08:44:10 on dist-test-slave-n6xs"
server_key: "25402f4e6c15d2214a549508445c6a84"
server_key_iv: "c0c2c011c3fbbb86910eaab09f1c38b1"
server_key_version: "encryptionkey@0"
I20260928 08:44:10.303658 3339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:10.319324 3339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:10.323622 3339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:10.323907 3339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:10.342245 3339 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38327
I20260928 08:44:10.342267 3405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38327 every 8 connection(s)
I20260928 08:44:10.343699 3339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:44:10.347348 3406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:10.348923 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 3339
I20260928 08:44:10.349150 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:44:10.349758 388 external_mini_cluster.cc:1428] Setting key 0f6a0564463ff80b607ebf226e7640ae
I20260928 08:44:10.361788 3406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 666de050916c4e4787b0b5130a8afeb1: Bootstrap starting.
Sep 28 08:44:10 dist-test-slave-n6xs krb5kdc[3326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:44:10.364878 3406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 666de050916c4e4787b0b5130a8afeb1: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:10.366226 3406 log.cc:824] T 00000000000000000000000000000000 P 666de050916c4e4787b0b5130a8afeb1: Log is configured to *not* fsync() on all Append() calls
I20260928 08:44:10.369423 3406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 666de050916c4e4787b0b5130a8afeb1: No bootstrap required, opened a new log
I20260928 08:44:10.372987 3409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:10.351505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43840 (local address 127.0.97.62:38327)
0928 08:44:10.352423 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:10.352441 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:10.352501 (+ 60us) server_negotiation.cc:407] Connection header received
0928 08:44:10.354280 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:10.354311 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:10.354814 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:10.355271 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:10.356652 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:10.358717 (+ 2065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:10.360000 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:10.361011 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:10.361189 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:10.364873 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:10.364884 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:10.364902 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:10.364958 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:10.368669 (+ 3711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:10.369293 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:10.369300 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:10.369309 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:10.369505 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:10.369896 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:10.369903 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:10.369907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:10.370593 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:10.370887 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:10.371176 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 08:44:10.371603 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":427,"threads_started":1}
I20260928 08:44:10.376618 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:44:10.378888 3406 raft_consensus.cc:348] T 00000000000000000000000000000000 P 666de050916c4e4787b0b5130a8afeb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666de050916c4e4787b0b5130a8afeb1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38327 } }
I20260928 08:44:10.379307 3406 raft_consensus.cc:374] T 00000000000000000000000000000000 P 666de050916c4e4787b0b5130a8afeb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:10.379410 3406 raft_consensus.cc:729] T 00000000000000000000000000000000 P 666de050916c4e4787b0b5130a8afeb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 666de050916c4e4787b0b5130a8afeb1, State: Initialized, Role: FOLLOWER
I20260928 08:44:10.380077 3406 consensus_queue.cc:260] T 00000000000000000000000000000000 P 666de050916c4e4787b0b5130a8afeb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666de050916c4e4787b0b5130a8afeb1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38327 } }
I20260928 08:44:10.380357 3406 raft_consensus.cc:388] T 00000000000000000000000000000000 P 666de050916c4e4787b0b5130a8afeb1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:44:10.380502 3406 raft_consensus.cc:482] T 00000000000000000000000000000000 P 666de050916c4e4787b0b5130a8afeb1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:44:10.380667 3406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 666de050916c4e4787b0b5130a8afeb1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:10.384100 3406 raft_consensus.cc:504] T 00000000000000000000000000000000 P 666de050916c4e4787b0b5130a8afeb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666de050916c4e4787b0b5130a8afeb1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38327 } }
I20260928 08:44:10.384588 3406 leader_election.cc:302] T 00000000000000000000000000000000 P 666de050916c4e4787b0b5130a8afeb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 666de050916c4e4787b0b5130a8afeb1; no voters:
I20260928 08:44:10.385223 3406 leader_election.cc:288] T 00000000000000000000000000000000 P 666de050916c4e4787b0b5130a8afeb1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:44:10.385314 3412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 666de050916c4e4787b0b5130a8afeb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:44:10.386595 3412 raft_consensus.cc:686] T 00000000000000000000000000000000 P 666de050916c4e4787b0b5130a8afeb1 [term 1 LEADER]: Becoming Leader. State: Replica: 666de050916c4e4787b0b5130a8afeb1, State: Running, Role: LEADER
I20260928 08:44:10.387197 3412 consensus_queue.cc:237] T 00000000000000000000000000000000 P 666de050916c4e4787b0b5130a8afeb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666de050916c4e4787b0b5130a8afeb1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38327 } }
I20260928 08:44:10.387534 3406 sys_catalog.cc:564] T 00000000000000000000000000000000 P 666de050916c4e4787b0b5130a8afeb1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:44:10.390232 3414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 666de050916c4e4787b0b5130a8afeb1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 666de050916c4e4787b0b5130a8afeb1. Latest consensus state: current_term: 1 leader_uuid: "666de050916c4e4787b0b5130a8afeb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666de050916c4e4787b0b5130a8afeb1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38327 } } }
I20260928 08:44:10.390659 3414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 666de050916c4e4787b0b5130a8afeb1 [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:10.390576 3413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 666de050916c4e4787b0b5130a8afeb1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "666de050916c4e4787b0b5130a8afeb1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "666de050916c4e4787b0b5130a8afeb1" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38327 } } }
I20260928 08:44:10.390949 3413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 666de050916c4e4787b0b5130a8afeb1 [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:10.398478 3421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:44:10.403837 3421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:44:10.413005 3421 catalog_manager.cc:1347] Generated new cluster ID: af77f68972fc4220980ab2f525e4e77b
I20260928 08:44:10.413112 3421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:44:10.426548 3409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:10.417840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43846 (local address 127.0.97.62:38327)
0928 08:44:10.418006 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:10.418012 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:10.418625 (+ 613us) server_negotiation.cc:407] Connection header received
0928 08:44:10.418812 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:10.418818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:10.418890 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:10.418993 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:10.420184 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:10.421263 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:10.423113 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:10.423785 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:10.423912 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:10.425023 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:10.425029 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:10.425035 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:44:10.425756 (+ 721us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:10.425823 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:44:10.425851 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:10.425944 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:10.426123 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 08:44:10.426222 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 08:44:10.469718 3421 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:44:10.471130 3421 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:44:10.480857 3421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 666de050916c4e4787b0b5130a8afeb1: Generated new TSK 0
I20260928 08:44:10.481772 3421 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:44:10.498742 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 3339
I20260928 08:44:10.499791 3401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:10.594800 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 3339
2026-09-28T08:44:10Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1943 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:44:10 dist-test-slave-n6xs krb5kdc[3440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:44:10 dist-test-slave-n6xs krb5kdc[3440](info): set up 2 sockets
Sep 28 08:44:10 dist-test-slave-n6xs krb5kdc[3440](info): commencing operation
krb5kdc: starting...
W20260928 08:44:11.676285 388 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.004s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:44:11 dist-test-slave-n6xs krb5kdc[3440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:44:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:44:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:44:11.867507 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45317
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45317
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:12.182679 3453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:12.182976 3453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:12.183075 3453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:12.190056 3453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:44:12.190163 3453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:12.190213 3453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:12.190263 3453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:44:12.190306 3453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:44:12.200169 3453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:12.201349 3453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:12.202598 3453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:12.213953 3461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:12.214068 3458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:12.214047 3459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:12.214659 3453 server_base.cc:1034] running on GCE node
I20260928 08:44:12.215340 3453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:12.216634 3453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:12.218192 3453 hybrid_clock.cc:648] HybridClock initialized: now 1790585052218159 us; error 47 us; skew 500 ppm
I20260928 08:44:12.218526 3453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:12 dist-test-slave-n6xs krb5kdc[3440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:12.221583 3453 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:44:12.222923 3453 webserver.cc:460] Webserver started at http://127.0.97.62:36215/ using document root <none> and password file <none>
I20260928 08:44:12.223639 3453 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:12.223783 3453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:12.224102 3453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:12.226917 3453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "da1350fcb3a54ba49027d79dd23976db"
format_stamp: "Formatted at 2026-09-28 08:44:12 on dist-test-slave-n6xs"
server_key: "cfe215aaeafe47dd94aacb3685fd05c4"
server_key_iv: "03b2ae1d2697d6192ac0ca6d457cf0fd"
server_key_version: "encryptionkey@0"
I20260928 08:44:12.227630 3453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "da1350fcb3a54ba49027d79dd23976db"
format_stamp: "Formatted at 2026-09-28 08:44:12 on dist-test-slave-n6xs"
server_key: "cfe215aaeafe47dd94aacb3685fd05c4"
server_key_iv: "03b2ae1d2697d6192ac0ca6d457cf0fd"
server_key_version: "encryptionkey@0"
I20260928 08:44:12.232604 3453 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 08:44:12.235786 3468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:12.237082 3453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:44:12.237257 3453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "da1350fcb3a54ba49027d79dd23976db"
format_stamp: "Formatted at 2026-09-28 08:44:12 on dist-test-slave-n6xs"
server_key: "cfe215aaeafe47dd94aacb3685fd05c4"
server_key_iv: "03b2ae1d2697d6192ac0ca6d457cf0fd"
server_key_version: "encryptionkey@0"
I20260928 08:44:12.237435 3453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:12.250588 3453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:12.254318 3453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:12.254594 3453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:12.273065 3453 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45317
I20260928 08:44:12.273095 3519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45317 every 8 connection(s)
I20260928 08:44:12.274655 3453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:44:12.278364 3520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:12.284518 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 3453
I20260928 08:44:12.284771 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:44:12.285392 388 external_mini_cluster.cc:1428] Setting key e5c83f80c0d46df7be80e11cafd72fee
I20260928 08:44:12.288416 3520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da1350fcb3a54ba49027d79dd23976db: Bootstrap starting.
I20260928 08:44:12.291306 3520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da1350fcb3a54ba49027d79dd23976db: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:12.292337 3520 log.cc:824] T 00000000000000000000000000000000 P da1350fcb3a54ba49027d79dd23976db: Log is configured to *not* fsync() on all Append() calls
I20260928 08:44:12.295017 3520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da1350fcb3a54ba49027d79dd23976db: No bootstrap required, opened a new log
Sep 28 08:44:12 dist-test-slave-n6xs krb5kdc[3440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:44:12.303179 3520 raft_consensus.cc:348] T 00000000000000000000000000000000 P da1350fcb3a54ba49027d79dd23976db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da1350fcb3a54ba49027d79dd23976db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45317 } }
I20260928 08:44:12.303568 3520 raft_consensus.cc:374] T 00000000000000000000000000000000 P da1350fcb3a54ba49027d79dd23976db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:12.303674 3520 raft_consensus.cc:729] T 00000000000000000000000000000000 P da1350fcb3a54ba49027d79dd23976db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da1350fcb3a54ba49027d79dd23976db, State: Initialized, Role: FOLLOWER
I20260928 08:44:12.304334 3520 consensus_queue.cc:260] T 00000000000000000000000000000000 P da1350fcb3a54ba49027d79dd23976db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da1350fcb3a54ba49027d79dd23976db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45317 } }
I20260928 08:44:12.304579 3520 raft_consensus.cc:388] T 00000000000000000000000000000000 P da1350fcb3a54ba49027d79dd23976db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:44:12.304715 3520 raft_consensus.cc:482] T 00000000000000000000000000000000 P da1350fcb3a54ba49027d79dd23976db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:44:12.304904 3520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da1350fcb3a54ba49027d79dd23976db [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:12.307986 3523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:12.287397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53926 (local address 127.0.97.62:45317)
0928 08:44:12.288187 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:12.288198 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:12.288235 (+ 37us) server_negotiation.cc:407] Connection header received
0928 08:44:12.289941 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:12.289963 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:12.290325 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:12.290688 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:12.292134 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:12.293796 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:12.295008 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:12.295741 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:12.295999 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:12.300097 (+ 4098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:12.300105 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:12.300123 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:12.300160 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:12.303269 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:12.304163 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:12.304177 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:12.304183 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:12.304346 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:12.304770 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:12.304775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:12.304779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:12.305490 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:12.305808 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:12.306159 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 08:44:12.306630 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":394,"threads_started":1}
I20260928 08:44:12.308135 3520 raft_consensus.cc:504] T 00000000000000000000000000000000 P da1350fcb3a54ba49027d79dd23976db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da1350fcb3a54ba49027d79dd23976db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45317 } }
I20260928 08:44:12.308714 3520 leader_election.cc:302] T 00000000000000000000000000000000 P da1350fcb3a54ba49027d79dd23976db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da1350fcb3a54ba49027d79dd23976db; no voters:
I20260928 08:44:12.309366 3520 leader_election.cc:288] T 00000000000000000000000000000000 P da1350fcb3a54ba49027d79dd23976db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:44:12.309504 3525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da1350fcb3a54ba49027d79dd23976db [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:44:12.311354 3520 sys_catalog.cc:564] T 00000000000000000000000000000000 P da1350fcb3a54ba49027d79dd23976db [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:44:12.311444 3525 raft_consensus.cc:686] T 00000000000000000000000000000000 P da1350fcb3a54ba49027d79dd23976db [term 1 LEADER]: Becoming Leader. State: Replica: da1350fcb3a54ba49027d79dd23976db, State: Running, Role: LEADER
I20260928 08:44:12.312026 3525 consensus_queue.cc:237] T 00000000000000000000000000000000 P da1350fcb3a54ba49027d79dd23976db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da1350fcb3a54ba49027d79dd23976db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45317 } }
I20260928 08:44:12.314438 3527 sys_catalog.cc:455] T 00000000000000000000000000000000 P da1350fcb3a54ba49027d79dd23976db [sys.catalog]: SysCatalogTable state changed. Reason: New leader da1350fcb3a54ba49027d79dd23976db. Latest consensus state: current_term: 1 leader_uuid: "da1350fcb3a54ba49027d79dd23976db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da1350fcb3a54ba49027d79dd23976db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45317 } } }
I20260928 08:44:12.314743 3527 sys_catalog.cc:458] T 00000000000000000000000000000000 P da1350fcb3a54ba49027d79dd23976db [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:12.315351 3526 sys_catalog.cc:455] T 00000000000000000000000000000000 P da1350fcb3a54ba49027d79dd23976db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da1350fcb3a54ba49027d79dd23976db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da1350fcb3a54ba49027d79dd23976db" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45317 } } }
I20260928 08:44:12.315603 3532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:44:12.315672 3526 sys_catalog.cc:458] T 00000000000000000000000000000000 P da1350fcb3a54ba49027d79dd23976db [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:12.321370 3532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:44:12.329550 3532 catalog_manager.cc:1347] Generated new cluster ID: 0eac245617e349d1ad0d7c0f3bd52fb9
I20260928 08:44:12.329668 3532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:44:12.339236 3532 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:44:12.341044 3532 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:44:12.349493 3532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da1350fcb3a54ba49027d79dd23976db: Generated new TSK 0
I20260928 08:44:12.350275 3532 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:44:12.403328 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45317
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:12.721844 3546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:12.722112 3546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:12.722203 3546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:12.729192 3546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:12.729285 3546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:12.729393 3546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:44:12.738513 3546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45317
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:12.739770 3546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:12.741115 3546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:12.753680 3551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:12.753612 3554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:12.753654 3552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:12.754787 3546 server_base.cc:1034] running on GCE node
I20260928 08:44:12.755319 3546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:12.756282 3546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:12.757540 3546 hybrid_clock.cc:648] HybridClock initialized: now 1790585052757477 us; error 32 us; skew 500 ppm
I20260928 08:44:12.757870 3546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:12 dist-test-slave-n6xs krb5kdc[3440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:12.761109 3546 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:44:12.762629 3546 webserver.cc:460] Webserver started at http://127.0.97.1:40959/ using document root <none> and password file <none>
I20260928 08:44:12.763299 3546 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:12.763425 3546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:12.763737 3546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:12.766584 3546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-0/data/instance:
uuid: "ea320028695948dbb75bdb3e4f4bf523"
format_stamp: "Formatted at 2026-09-28 08:44:12 on dist-test-slave-n6xs"
server_key: "1d1fcd0f83042b5262a92939b5fba449"
server_key_iv: "c0a763b1b5a5ebe938b3620d056affab"
server_key_version: "encryptionkey@0"
I20260928 08:44:12.767344 3546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ea320028695948dbb75bdb3e4f4bf523"
format_stamp: "Formatted at 2026-09-28 08:44:12 on dist-test-slave-n6xs"
server_key: "1d1fcd0f83042b5262a92939b5fba449"
server_key_iv: "c0a763b1b5a5ebe938b3620d056affab"
server_key_version: "encryptionkey@0"
I20260928 08:44:12.772395 3546 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 08:44:12.775645 3561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:12.777522 3546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:44:12.777689 3546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-0/wal
uuid: "ea320028695948dbb75bdb3e4f4bf523"
format_stamp: "Formatted at 2026-09-28 08:44:12 on dist-test-slave-n6xs"
server_key: "1d1fcd0f83042b5262a92939b5fba449"
server_key_iv: "c0a763b1b5a5ebe938b3620d056affab"
server_key_version: "encryptionkey@0"
I20260928 08:44:12.777875 3546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:12.800812 3546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:12.805081 3546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:12.805379 3546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:12.806329 3546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:44:12.807724 3546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:44:12.807847 3546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:12.807993 3546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:44:12.808063 3546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:12.841262 3546 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40943
I20260928 08:44:12.841303 3673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40943 every 8 connection(s)
I20260928 08:44:12.842512 3546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:44:12.850906 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 3546
I20260928 08:44:12.851154 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:44:12.851994 388 external_mini_cluster.cc:1428] Setting key 3735e725a92e0178488303139fd18e63
Sep 28 08:44:12 dist-test-slave-n6xs krb5kdc[3440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:44:12.863405 3676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:12.845109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54305 (local address 127.0.97.62:45317)
0928 08:44:12.845615 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:12.845620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:12.846616 (+ 996us) server_negotiation.cc:407] Connection header received
0928 08:44:12.848452 (+ 1836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:12.848458 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:12.848557 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:12.848673 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:12.850133 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:12.850863 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:12.853028 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:12.853484 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:12.853613 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:12.857097 (+ 3484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:12.857106 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:12.857114 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:12.857159 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:12.860202 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:12.861144 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:12.861149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:12.861151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:12.861271 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:12.861743 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:12.861746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:12.861748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:12.861999 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:12.862105 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:12.862972 (+ 867us) server_negotiation.cc:299] Negotiation successful
0928 08:44:12.863143 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":395,"thread_start_us":285,"threads_started":1}
I20260928 08:44:12.864892 3677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:12.845723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45317 (local address 127.0.97.1:54305)
0928 08:44:12.846448 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:12.846491 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:12.848181 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:12.848860 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:12.848868 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:12.849353 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:12.849928 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:12.849945 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:12.851014 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:12.851021 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:12.852895 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:12.852914 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:12.853727 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:12.853735 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:12.854015 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:12.854688 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:12.854710 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:12.856868 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:12.860329 (+ 3461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:12.860341 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:12.860360 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:12.860990 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:12.861385 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:12.861392 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:12.861398 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:12.861631 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:12.862128 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:12.862139 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:12.862573 (+ 434us) client_negotiation.cc:769] Sending connection context
0928 08:44:12.862866 (+ 293us) client_negotiation.cc:240] Negotiation successful
0928 08:44:12.863422 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":288,"threads_started":1}
I20260928 08:44:12.866464 3674 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45317
I20260928 08:44:12.866802 3674 heartbeater.cc:464] Registering TS with master...
I20260928 08:44:12.867486 3674 heartbeater.cc:511] Master 127.0.97.62:45317 requested a full tablet report, sending...
I20260928 08:44:12.869621 3485 ts_manager.cc:194] Registered new tserver with Master: ea320028695948dbb75bdb3e4f4bf523 (127.0.97.1:40943)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 08:44:12.871065 3485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54305
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 08:44:12.895864 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45317
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:13.217108 3680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:13.217417 3680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:13.217605 3680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:13.225071 3680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:13.225220 3680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:13.225387 3680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:44:13.234987 3680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45317
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:13.236318 3680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:13.237603 3680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:13.250473 3686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:13.250524 3688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:13.250854 3685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:13.251258 3680 server_base.cc:1034] running on GCE node
I20260928 08:44:13.251960 3680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:13.252810 3680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:13.254081 3680 hybrid_clock.cc:648] HybridClock initialized: now 1790585053254023 us; error 54 us; skew 500 ppm
I20260928 08:44:13.254380 3680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:13 dist-test-slave-n6xs krb5kdc[3440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:13.257423 3680 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:44:13.258781 3680 webserver.cc:460] Webserver started at http://127.0.97.2:43951/ using document root <none> and password file <none>
I20260928 08:44:13.259408 3680 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:13.259500 3680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:13.259840 3680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:13.262863 3680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-1/data/instance:
uuid: "95d9b6ef5d7444498d7fa14221402351"
format_stamp: "Formatted at 2026-09-28 08:44:13 on dist-test-slave-n6xs"
server_key: "5608bbd9bf3f4181dce64bedc3b1c493"
server_key_iv: "e2be0cff036642262ed04bf7de7f8fb5"
server_key_version: "encryptionkey@0"
I20260928 08:44:13.263670 3680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-1/wal/instance:
uuid: "95d9b6ef5d7444498d7fa14221402351"
format_stamp: "Formatted at 2026-09-28 08:44:13 on dist-test-slave-n6xs"
server_key: "5608bbd9bf3f4181dce64bedc3b1c493"
server_key_iv: "e2be0cff036642262ed04bf7de7f8fb5"
server_key_version: "encryptionkey@0"
I20260928 08:44:13.268567 3680 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 08:44:13.271936 3695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:13.273527 3680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 08:44:13.273679 3680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-1/wal
uuid: "95d9b6ef5d7444498d7fa14221402351"
format_stamp: "Formatted at 2026-09-28 08:44:13 on dist-test-slave-n6xs"
server_key: "5608bbd9bf3f4181dce64bedc3b1c493"
server_key_iv: "e2be0cff036642262ed04bf7de7f8fb5"
server_key_version: "encryptionkey@0"
I20260928 08:44:13.273888 3680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:13.296550 3680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:13.300817 3680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:13.301101 3680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:13.301990 3680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:44:13.303359 3680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:44:13.303438 3680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:13.303531 3680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:44:13.303581 3680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:13.339190 3680 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39321
I20260928 08:44:13.339218 3807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39321 every 8 connection(s)
I20260928 08:44:13.340575 3680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:44:13.344947 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 3680
I20260928 08:44:13.345134 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:44:13.345686 388 external_mini_cluster.cc:1428] Setting key 7c2291f395156babf6cc61c7e99beeb9
Sep 28 08:44:13 dist-test-slave-n6xs krb5kdc[3440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:44:13.363265 3676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:13.343166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49915 (local address 127.0.97.62:45317)
0928 08:44:13.343376 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:13.343381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:13.344674 (+ 1293us) server_negotiation.cc:407] Connection header received
0928 08:44:13.346892 (+ 2218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:13.346897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:13.346967 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:13.347080 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:13.348992 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:13.349952 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:13.352403 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:13.352788 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:13.352879 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:13.356571 (+ 3692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:13.356581 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:13.356589 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:13.356632 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:13.360101 (+ 3469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:13.360802 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:13.360810 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:13.360817 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:13.360978 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:13.361406 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:13.361413 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:13.361418 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:13.361822 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:13.361988 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:13.362630 (+ 642us) server_negotiation.cc:299] Negotiation successful
0928 08:44:13.362889 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260928 08:44:13.363988 3810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:13.343679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45317 (local address 127.0.97.2:49915)
0928 08:44:13.344457 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:13.344501 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:13.346649 (+ 2148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:13.347279 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:13.347287 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:13.347989 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:13.348796 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:13.348824 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:13.350112 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:13.350120 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:13.352269 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:13.352286 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:13.353004 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:13.353031 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:13.353316 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:13.354005 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:13.354031 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:13.356326 (+ 2295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:13.360264 (+ 3938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:13.360270 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:13.360282 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:13.360675 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:13.361100 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:13.361105 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:13.361109 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:13.361282 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:13.361950 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:13.361955 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:13.362248 (+ 293us) client_negotiation.cc:769] Sending connection context
0928 08:44:13.362473 (+ 225us) client_negotiation.cc:240] Negotiation successful
0928 08:44:13.362845 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":345,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260928 08:44:13.365473 3808 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45317
I20260928 08:44:13.365783 3808 heartbeater.cc:464] Registering TS with master...
I20260928 08:44:13.366541 3808 heartbeater.cc:511] Master 127.0.97.62:45317 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260928 08:44:13.367992 3485 ts_manager.cc:194] Registered new tserver with Master: 95d9b6ef5d7444498d7fa14221402351 (127.0.97.2:39321)
I20260928 08:44:13.368752 3485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:49915
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:44:13.390913 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45317
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:38795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:13.702608 3813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:13.702886 3813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:13.703050 3813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:13.710129 3813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:13.710228 3813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:13.710404 3813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:44:13.720285 3813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45317
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:13.721549 3813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:13.722788 3813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:13.736222 3818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:13.736191 3821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:13.736286 3819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:13.737437 3813 server_base.cc:1034] running on GCE node
I20260928 08:44:13.738029 3813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:13.738956 3813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:13.740146 3813 hybrid_clock.cc:648] HybridClock initialized: now 1790585053740083 us; error 55 us; skew 500 ppm
I20260928 08:44:13.740456 3813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:13 dist-test-slave-n6xs krb5kdc[3440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:13.743590 3813 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:44:13.745344 3813 webserver.cc:460] Webserver started at http://127.0.97.3:46081/ using document root <none> and password file <none>
I20260928 08:44:13.746218 3813 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:13.746455 3813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:13.746784 3813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:13.749691 3813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-2/data/instance:
uuid: "05219a0e38da4098886286bdf47a0b64"
format_stamp: "Formatted at 2026-09-28 08:44:13 on dist-test-slave-n6xs"
server_key: "0e724fad09e0eb3b4310b2b091a858a5"
server_key_iv: "e782a954588a6ad81f7495383ad2dc2b"
server_key_version: "encryptionkey@0"
I20260928 08:44:13.750452 3813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-2/wal/instance:
uuid: "05219a0e38da4098886286bdf47a0b64"
format_stamp: "Formatted at 2026-09-28 08:44:13 on dist-test-slave-n6xs"
server_key: "0e724fad09e0eb3b4310b2b091a858a5"
server_key_iv: "e782a954588a6ad81f7495383ad2dc2b"
server_key_version: "encryptionkey@0"
I20260928 08:44:13.755390 3813 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 08:44:13.758862 3828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:13.760424 3813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:44:13.760618 3813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-2/wal
uuid: "05219a0e38da4098886286bdf47a0b64"
format_stamp: "Formatted at 2026-09-28 08:44:13 on dist-test-slave-n6xs"
server_key: "0e724fad09e0eb3b4310b2b091a858a5"
server_key_iv: "e782a954588a6ad81f7495383ad2dc2b"
server_key_version: "encryptionkey@0"
I20260928 08:44:13.760797 3813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:13.775385 3813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:13.779911 3813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:13.780205 3813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:13.781148 3813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:44:13.782464 3813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:44:13.782531 3813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:13.782701 3813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:44:13.782750 3813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:13.816115 3813 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43815
I20260928 08:44:13.816166 3940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43815 every 8 connection(s)
I20260928 08:44:13.817414 3813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:44:13.818059 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 3813
I20260928 08:44:13.818276 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790585022442112-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:44:13.818892 388 external_mini_cluster.cc:1428] Setting key 2458658723cac111693a989abb82728f
Sep 28 08:44:13 dist-test-slave-n6xs krb5kdc[3440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585053, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:44:13.837090 3676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:13.820412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:32981 (local address 127.0.97.62:45317)
0928 08:44:13.820616 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:13.820621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:13.822176 (+ 1555us) server_negotiation.cc:407] Connection header received
0928 08:44:13.823955 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:13.823959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:13.824019 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:13.824097 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:13.825438 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:13.826172 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:13.827681 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:13.828092 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:13.828184 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:13.831472 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:13.831477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:13.831480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:13.831511 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:13.834599 (+ 3088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:13.835186 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:13.835189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:13.835192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:13.835308 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:13.835689 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:13.835692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:13.835694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:13.835998 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:13.836111 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:13.836695 (+ 584us) server_negotiation.cc:299] Negotiation successful
0928 08:44:13.836844 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 08:44:13.838012 3943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:13.821100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45317 (local address 127.0.97.3:32981)
0928 08:44:13.822015 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:13.822061 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:13.823678 (+ 1617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:13.824264 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:13.824272 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:13.824742 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:13.825304 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:13.825319 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:13.826293 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:13.826297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:13.827559 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:13.827570 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:13.828264 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:13.828269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:13.828499 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:13.829230 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:13.829251 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:13.831297 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:13.834698 (+ 3401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:13.834704 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:13.834716 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:13.835093 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:13.835427 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:13.835431 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:13.835433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:13.835599 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:13.836099 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:13.836104 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:13.836400 (+ 296us) client_negotiation.cc:769] Sending connection context
0928 08:44:13.836613 (+ 213us) client_negotiation.cc:240] Negotiation successful
0928 08:44:13.836961 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":387,"threads_started":1}
I20260928 08:44:13.839236 3941 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45317
I20260928 08:44:13.839540 3941 heartbeater.cc:464] Registering TS with master...
I20260928 08:44:13.840375 3941 heartbeater.cc:511] Master 127.0.97.62:45317 requested a full tablet report, sending...
I20260928 08:44:13.841786 3485 ts_manager.cc:194] Registered new tserver with Master: 05219a0e38da4098886286bdf47a0b64 (127.0.97.3:43815)
I20260928 08:44:13.842628 3485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:32981
I20260928 08:44:13.846650 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 08:44:13 dist-test-slave-n6xs krb5kdc[3440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585053, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
I20260928 08:44:13.874218 3674 heartbeater.cc:503] Master 127.0.97.62:45317 was elected leader, sending a full tablet report...
Sep 28 08:44:13 dist-test-slave-n6xs krb5kdc[3440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585053, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:44:13.893085 3676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:13.879394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49842 (local address 127.0.97.62:45317)
0928 08:44:13.879558 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:13.879562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:13.880013 (+ 451us) server_negotiation.cc:407] Connection header received
0928 08:44:13.880138 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:13.880142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:13.880204 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:13.880357 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:13.881731 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:13.882529 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:13.883790 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:13.884256 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:13.884352 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:13.888048 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:13.888053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:13.888056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:13.888087 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:13.890814 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:13.891347 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:13.891351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:13.891353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:13.891471 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:13.891867 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:13.891874 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:13.891878 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:13.892196 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:13.892386 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:13.892675 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 08:44:13.892859 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 08:44:13.893692 3485 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49842
I20260928 08:44:13.895915 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 3546
Sep 28 08:44:13 dist-test-slave-n6xs krb5kdc[3440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585053, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:44:13.912225 3952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:13.896802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47966 (local address 127.0.97.1:40943)
0928 08:44:13.897436 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:13.897444 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:13.897473 (+ 29us) server_negotiation.cc:407] Connection header received
0928 08:44:13.897525 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:13.897532 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:13.897739 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:13.897904 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:13.899011 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:13.900116 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:13.901300 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:13.901839 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:13.901985 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:13.905726 (+ 3741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:13.905736 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:13.905749 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:13.905795 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:13.909115 (+ 3320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:13.909954 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:13.909963 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:13.909966 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:13.910108 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:13.910572 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:13.910578 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:13.910582 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:13.911069 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:13.911301 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:13.911630 (+ 329us) server_negotiation.cc:299] Negotiation successful
0928 08:44:13.911891 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":254,"threads_started":1}
W20260928 08:44:13.913421 3669 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47966
W20260928 08:44:13.914677 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 3546: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:44:13.914851 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 3546
I20260928 08:44:13.930354 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 3680
Sep 28 08:44:13 dist-test-slave-n6xs krb5kdc[3440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585053, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 08:44:13.947201 3953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:13.931125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51618 (local address 127.0.97.2:39321)
0928 08:44:13.931607 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:13.931614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:13.931638 (+ 24us) server_negotiation.cc:407] Connection header received
0928 08:44:13.931693 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:13.931700 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:13.931972 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:13.932097 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:13.933257 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:13.934281 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:13.935785 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:13.936533 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:13.936636 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:13.941327 (+ 4691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:13.941337 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:13.941352 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:13.941386 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:13.944561 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:13.945215 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:13.945222 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:13.945224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:13.945357 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:13.945751 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:13.945754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:13.945756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:13.946081 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:13.946293 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:13.946716 (+ 423us) server_negotiation.cc:299] Negotiation successful
0928 08:44:13.946924 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":251,"threads_started":1}
W20260928 08:44:13.948562 3803 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51618
W20260928 08:44:13.949680 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 3680: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:44:13.949829 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 3680
I20260928 08:44:13.965355 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 3813
Sep 28 08:44:13 dist-test-slave-n6xs krb5kdc[3440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585053, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 08:44:13.982846 3954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:13.966508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47248 (local address 127.0.97.3:43815)
0928 08:44:13.967163 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:13.967170 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:13.967199 (+ 29us) server_negotiation.cc:407] Connection header received
0928 08:44:13.967256 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:13.967262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:13.967526 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:13.967656 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:13.969026 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:13.970099 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:13.971303 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:13.972076 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:13.972205 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:13.976977 (+ 4772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:13.976990 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:13.977005 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:13.977040 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:13.980191 (+ 3151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:13.980923 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:13.980933 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:13.980937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:13.981069 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:13.981367 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:13.981370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:13.981373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:13.981709 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:13.981957 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:13.982195 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 08:44:13.982404 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":361,"threads_started":1}
W20260928 08:44:13.984292 3936 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47248
W20260928 08:44:13.985519 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 3813: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:44:13.985646 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 3813
I20260928 08:44:14.000916 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 3453
I20260928 08:44:14.002285 3515 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:14.103633 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 3453
2026-09-28T08:44:14Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3513 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:44:14 dist-test-slave-n6xs krb5kdc[3958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:44:14 dist-test-slave-n6xs krb5kdc[3958](info): set up 2 sockets
Sep 28 08:44:14 dist-test-slave-n6xs krb5kdc[3958](info): commencing operation
krb5kdc: starting...
W20260928 08:44:15.180313 388 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.003s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:44:15 dist-test-slave-n6xs krb5kdc[3958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:44:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:44:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:44:15.381150 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39119
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39119
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:15.704582 3971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:15.704950 3971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:15.705108 3971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:15.712401 3971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:44:15.712538 3971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:15.712656 3971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:15.712751 3971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:44:15.712836 3971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:44:15.722618 3971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:15.724012 3971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:15.725250 3971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:15.736948 3976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:15.736996 3977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:15.737021 3979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:15.738452 3971 server_base.cc:1034] running on GCE node
I20260928 08:44:15.739259 3971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:15.740856 3971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:15.742133 3971 hybrid_clock.cc:648] HybridClock initialized: now 1790585055742085 us; error 56 us; skew 500 ppm
I20260928 08:44:15.742529 3971 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:15 dist-test-slave-n6xs krb5kdc[3958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:15.749886 3971 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:44:15.751546 3971 webserver.cc:460] Webserver started at http://127.0.97.62:36441/ using document root <none> and password file <none>
I20260928 08:44:15.752466 3971 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:15.752625 3971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:15.753051 3971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:15.756199 3971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "cb6b6c6e8c4540a9a9af183ed9444e94"
format_stamp: "Formatted at 2026-09-28 08:44:15 on dist-test-slave-n6xs"
server_key: "b309e104162c20abf7d92d864c660281"
server_key_iv: "83ef33e98cc7f4f25919eb8a7bf37e17"
server_key_version: "encryptionkey@0"
I20260928 08:44:15.756995 3971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "cb6b6c6e8c4540a9a9af183ed9444e94"
format_stamp: "Formatted at 2026-09-28 08:44:15 on dist-test-slave-n6xs"
server_key: "b309e104162c20abf7d92d864c660281"
server_key_iv: "83ef33e98cc7f4f25919eb8a7bf37e17"
server_key_version: "encryptionkey@0"
I20260928 08:44:15.762096 3971 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:44:15.765556 3986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:15.767074 3971 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:44:15.767257 3971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "cb6b6c6e8c4540a9a9af183ed9444e94"
format_stamp: "Formatted at 2026-09-28 08:44:15 on dist-test-slave-n6xs"
server_key: "b309e104162c20abf7d92d864c660281"
server_key_iv: "83ef33e98cc7f4f25919eb8a7bf37e17"
server_key_version: "encryptionkey@0"
I20260928 08:44:15.767486 3971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:15.794037 3971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:15.798457 3971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:15.798765 3971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:15.817677 3971 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39119
I20260928 08:44:15.817720 4037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39119 every 8 connection(s)
I20260928 08:44:15.819056 3971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:44:15.819969 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 3971
I20260928 08:44:15.820196 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:44:15.820935 388 external_mini_cluster.cc:1428] Setting key 9923cb2e3c060a81ddf307ac664c28ab
I20260928 08:44:15.823374 4038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:15.834579 4038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb6b6c6e8c4540a9a9af183ed9444e94: Bootstrap starting.
Sep 28 08:44:15 dist-test-slave-n6xs krb5kdc[3958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:44:15.838160 4038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb6b6c6e8c4540a9a9af183ed9444e94: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:15.839375 4038 log.cc:824] T 00000000000000000000000000000000 P cb6b6c6e8c4540a9a9af183ed9444e94: Log is configured to *not* fsync() on all Append() calls
I20260928 08:44:15.842739 4038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb6b6c6e8c4540a9a9af183ed9444e94: No bootstrap required, opened a new log
I20260928 08:44:15.844735 4041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:15.822988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38686 (local address 127.0.97.62:39119)
0928 08:44:15.824035 (+ 1047us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:15.824047 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:15.824091 (+ 44us) server_negotiation.cc:407] Connection header received
0928 08:44:15.826106 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:15.826129 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:15.826551 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:15.826980 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:15.828624 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:15.829987 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:15.831274 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:15.832105 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:15.832341 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:15.836696 (+ 4355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:15.836704 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:15.836719 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:15.836755 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:15.840483 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:15.841418 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:15.841424 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:15.841430 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:15.841577 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:15.841988 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:15.841992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:15.841993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:15.842520 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:15.842719 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:15.843197 (+ 478us) server_negotiation.cc:299] Negotiation successful
0928 08:44:15.843628 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":826,"thread_start_us":544,"threads_started":1}
I20260928 08:44:15.850975 4038 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb6b6c6e8c4540a9a9af183ed9444e94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6b6c6e8c4540a9a9af183ed9444e94" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39119 } }
I20260928 08:44:15.851349 4038 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb6b6c6e8c4540a9a9af183ed9444e94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:15.851471 4038 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb6b6c6e8c4540a9a9af183ed9444e94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6b6c6e8c4540a9a9af183ed9444e94, State: Initialized, Role: FOLLOWER
I20260928 08:44:15.852085 4038 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb6b6c6e8c4540a9a9af183ed9444e94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6b6c6e8c4540a9a9af183ed9444e94" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39119 } }
I20260928 08:44:15.852301 4038 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb6b6c6e8c4540a9a9af183ed9444e94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:44:15.852423 4038 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb6b6c6e8c4540a9a9af183ed9444e94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:44:15.852609 4038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb6b6c6e8c4540a9a9af183ed9444e94 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:15.854875 4038 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb6b6c6e8c4540a9a9af183ed9444e94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6b6c6e8c4540a9a9af183ed9444e94" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39119 } }
I20260928 08:44:15.855362 4038 leader_election.cc:302] T 00000000000000000000000000000000 P cb6b6c6e8c4540a9a9af183ed9444e94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb6b6c6e8c4540a9a9af183ed9444e94; no voters:
I20260928 08:44:15.856076 4038 leader_election.cc:288] T 00000000000000000000000000000000 P cb6b6c6e8c4540a9a9af183ed9444e94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:44:15.856205 4043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb6b6c6e8c4540a9a9af183ed9444e94 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:44:15.856972 4043 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb6b6c6e8c4540a9a9af183ed9444e94 [term 1 LEADER]: Becoming Leader. State: Replica: cb6b6c6e8c4540a9a9af183ed9444e94, State: Running, Role: LEADER
I20260928 08:44:15.857532 4043 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb6b6c6e8c4540a9a9af183ed9444e94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6b6c6e8c4540a9a9af183ed9444e94" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39119 } }
I20260928 08:44:15.858284 4038 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb6b6c6e8c4540a9a9af183ed9444e94 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:44:15.860594 4044 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb6b6c6e8c4540a9a9af183ed9444e94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb6b6c6e8c4540a9a9af183ed9444e94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6b6c6e8c4540a9a9af183ed9444e94" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39119 } } }
I20260928 08:44:15.860966 4044 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb6b6c6e8c4540a9a9af183ed9444e94 [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:15.861486 4045 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb6b6c6e8c4540a9a9af183ed9444e94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb6b6c6e8c4540a9a9af183ed9444e94. Latest consensus state: current_term: 1 leader_uuid: "cb6b6c6e8c4540a9a9af183ed9444e94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6b6c6e8c4540a9a9af183ed9444e94" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39119 } } }
I20260928 08:44:15.861783 4045 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb6b6c6e8c4540a9a9af183ed9444e94 [sys.catalog]: This master's current role is: LEADER
W20260928 08:44:15.868391 4059 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cb6b6c6e8c4540a9a9af183ed9444e94: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:44:15.868494 4059 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:44:15.868624 4052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:44:15.870720 4052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:44:15.879623 4052 catalog_manager.cc:1347] Generated new cluster ID: 903d3d3713f14d7780c4d6a0b71a6a6a
I20260928 08:44:15.879768 4052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:44:15.890501 4052 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:44:15.891897 4052 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:44:15.907666 4052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb6b6c6e8c4540a9a9af183ed9444e94: Generated new TSK 0
I20260928 08:44:15.908576 4052 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:44:15.957520 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39119
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:16.283309 4064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:16.283595 4064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:16.283748 4064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:16.291006 4064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:16.291113 4064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:16.291304 4064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:44:16.301134 4064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:16.302542 4064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:16.303890 4064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:16.316915 4072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:16.316865 4070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:16.316954 4069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:16.317693 4064 server_base.cc:1034] running on GCE node
I20260928 08:44:16.318295 4064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:16.319270 4064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:16.320490 4064 hybrid_clock.cc:648] HybridClock initialized: now 1790585056320446 us; error 49 us; skew 500 ppm
I20260928 08:44:16.320799 4064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:16 dist-test-slave-n6xs krb5kdc[3958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:16.323869 4064 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:44:16.325364 4064 webserver.cc:460] Webserver started at http://127.0.97.1:38311/ using document root <none> and password file <none>
I20260928 08:44:16.326089 4064 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:16.326234 4064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:16.326542 4064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:16.329533 4064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-0/data/instance:
uuid: "5c0d2f4f99934a39a70f22d69396860c"
format_stamp: "Formatted at 2026-09-28 08:44:16 on dist-test-slave-n6xs"
server_key: "18a4823b9a6400e8d403b05a1587a16f"
server_key_iv: "a78b288a73596643b7d39c72bb121265"
server_key_version: "encryptionkey@0"
I20260928 08:44:16.330295 4064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5c0d2f4f99934a39a70f22d69396860c"
format_stamp: "Formatted at 2026-09-28 08:44:16 on dist-test-slave-n6xs"
server_key: "18a4823b9a6400e8d403b05a1587a16f"
server_key_iv: "a78b288a73596643b7d39c72bb121265"
server_key_version: "encryptionkey@0"
I20260928 08:44:16.335342 4064 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 08:44:16.338747 4079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:16.340107 4064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:44:16.340266 4064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-0/wal
uuid: "5c0d2f4f99934a39a70f22d69396860c"
format_stamp: "Formatted at 2026-09-28 08:44:16 on dist-test-slave-n6xs"
server_key: "18a4823b9a6400e8d403b05a1587a16f"
server_key_iv: "a78b288a73596643b7d39c72bb121265"
server_key_version: "encryptionkey@0"
I20260928 08:44:16.340476 4064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:16.370887 4064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:16.375047 4064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:16.375329 4064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:16.376416 4064 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:44:16.377676 4064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:44:16.377781 4064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:16.377923 4064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:44:16.378008 4064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:16.410473 4064 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37877
I20260928 08:44:16.410560 4191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37877 every 8 connection(s)
I20260928 08:44:16.411897 4064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:44:16.417239 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 4064
I20260928 08:44:16.417456 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:44:16.418234 388 external_mini_cluster.cc:1428] Setting key 328ea811b04e2ac2fe299a703fad8b45
Sep 28 08:44:16 dist-test-slave-n6xs krb5kdc[3958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:44:16.434299 4194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:16.414391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42729 (local address 127.0.97.62:39119)
0928 08:44:16.414815 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:16.414819 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:16.415851 (+ 1032us) server_negotiation.cc:407] Connection header received
0928 08:44:16.418136 (+ 2285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:16.418140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:16.418204 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:16.418364 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:16.420158 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:16.420885 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:16.423433 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:16.423934 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:16.424030 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:16.428431 (+ 4401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:16.428436 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:16.428439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:16.428483 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:16.430898 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:16.431792 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:16.431796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:16.431798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:16.431958 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:16.432491 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:16.432494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:16.432496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:16.432733 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:16.432837 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:16.433757 (+ 920us) server_negotiation.cc:299] Negotiation successful
0928 08:44:16.433933 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":203,"threads_started":1}
I20260928 08:44:16.435890 4195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:16.415006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39119 (local address 127.0.97.1:42729)
0928 08:44:16.415676 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:16.415718 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:16.417813 (+ 2095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:16.418620 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:16.418630 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:16.419354 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:16.419990 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:16.420024 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:16.421048 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:16.421056 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:16.423274 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:16.423292 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:16.424145 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:16.424154 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:16.424471 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:16.425156 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:16.425187 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:16.428172 (+ 2985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:16.431035 (+ 2863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:16.431047 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:16.431066 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:16.431668 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:16.432064 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:16.432071 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:16.432078 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:16.432356 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:16.432865 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:16.432876 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:16.433346 (+ 470us) client_negotiation.cc:769] Sending connection context
0928 08:44:16.433652 (+ 306us) client_negotiation.cc:240] Negotiation successful
0928 08:44:16.434251 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":284,"threads_started":1}
I20260928 08:44:16.437671 4192 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39119
I20260928 08:44:16.438005 4192 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 08:44:16.438854 4192 heartbeater.cc:511] Master 127.0.97.62:39119 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260928 08:44:16.441413 4003 ts_manager.cc:194] Registered new tserver with Master: 5c0d2f4f99934a39a70f22d69396860c (127.0.97.1:37877)
I20260928 08:44:16.443667 4003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:42729
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 08:44:16.466917 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39119
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:16.788731 4198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:16.789012 4198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:16.789186 4198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:16.796288 4198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:16.796388 4198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:16.796574 4198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:44:16.806188 4198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:16.807504 4198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:16.808794 4198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:16.822325 4204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:16.822573 4206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:16.822815 4203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:16.823592 4198 server_base.cc:1034] running on GCE node
I20260928 08:44:16.824246 4198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:16.825222 4198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:16.826431 4198 hybrid_clock.cc:648] HybridClock initialized: now 1790585056826383 us; error 42 us; skew 500 ppm
I20260928 08:44:16.826802 4198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:16 dist-test-slave-n6xs krb5kdc[3958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:16.829916 4198 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:44:16.831460 4198 webserver.cc:460] Webserver started at http://127.0.97.2:33043/ using document root <none> and password file <none>
I20260928 08:44:16.832185 4198 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:16.832319 4198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:16.832623 4198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:16.835377 4198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-1/data/instance:
uuid: "5c7223cd519f40bc8da8e4b71a323349"
format_stamp: "Formatted at 2026-09-28 08:44:16 on dist-test-slave-n6xs"
server_key: "a2a9b99a75a0c9f4eff40c1535286353"
server_key_iv: "aeebfbd25a2af9935a4c50b1b7483bec"
server_key_version: "encryptionkey@0"
I20260928 08:44:16.836210 4198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-1/wal/instance:
uuid: "5c7223cd519f40bc8da8e4b71a323349"
format_stamp: "Formatted at 2026-09-28 08:44:16 on dist-test-slave-n6xs"
server_key: "a2a9b99a75a0c9f4eff40c1535286353"
server_key_iv: "aeebfbd25a2af9935a4c50b1b7483bec"
server_key_version: "encryptionkey@0"
I20260928 08:44:16.841269 4198 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 08:44:16.844890 4213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:16.846289 4198 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 08:44:16.846500 4198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-1/wal
uuid: "5c7223cd519f40bc8da8e4b71a323349"
format_stamp: "Formatted at 2026-09-28 08:44:16 on dist-test-slave-n6xs"
server_key: "a2a9b99a75a0c9f4eff40c1535286353"
server_key_iv: "aeebfbd25a2af9935a4c50b1b7483bec"
server_key_version: "encryptionkey@0"
I20260928 08:44:16.846697 4198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:16.862671 4198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:16.867282 4198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:16.867604 4198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:16.868660 4198 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:44:16.870062 4198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:44:16.870141 4198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:16.870322 4198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:44:16.870380 4198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:16.907411 4198 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44415
I20260928 08:44:16.907526 4325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44415 every 8 connection(s)
I20260928 08:44:16.908905 4198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:44:16.915194 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 4198
I20260928 08:44:16.915375 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:44:16.915799 388 external_mini_cluster.cc:1428] Setting key 888393b05f8ae3dec5de263f1f024979
Sep 28 08:44:16 dist-test-slave-n6xs krb5kdc[3958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585056, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260928 08:44:16.936087 4194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:16.911568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58279 (local address 127.0.97.62:39119)
0928 08:44:16.911779 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:16.911784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:16.913146 (+ 1362us) server_negotiation.cc:407] Connection header received
0928 08:44:16.915063 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:16.915069 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:16.915161 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:16.915357 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:16.917287 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:16.918218 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:16.920232 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:16.921003 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:16.921154 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:16.929555 (+ 8401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:16.929561 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:16.929565 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:16.929602 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:16.932500 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:16.933384 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:16.933389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:16.933391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:16.933519 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:16.934022 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:16.934032 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:16.934034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:16.934326 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:16.934509 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:16.935391 (+ 882us) server_negotiation.cc:299] Negotiation successful
0928 08:44:16.935675 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260928 08:44:16.937474 4328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:16.912153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39119 (local address 127.0.97.2:58279)
0928 08:44:16.912947 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:16.912996 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:16.914742 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:16.915614 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:16.915623 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:16.916386 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:16.917097 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:16.917116 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:16.918395 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:16.918406 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:16.919996 (+ 1590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:16.920012 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:16.921336 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:16.921348 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:16.921755 (+ 407us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:16.926280 (+ 4525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:16.926314 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:16.929252 (+ 2938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:16.932685 (+ 3433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:16.932695 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:16.932716 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:16.933217 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:16.933648 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:16.933657 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:16.933661 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:16.933887 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:16.934481 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:16.934490 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:16.934962 (+ 472us) client_negotiation.cc:769] Sending connection context
0928 08:44:16.935259 (+ 297us) client_negotiation.cc:240] Negotiation successful
0928 08:44:16.935888 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":306,"threads_started":1}
I20260928 08:44:16.939201 4326 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39119
I20260928 08:44:16.939491 4326 heartbeater.cc:464] Registering TS with master...
I20260928 08:44:16.940323 4326 heartbeater.cc:511] Master 127.0.97.62:39119 requested a full tablet report, sending...
I20260928 08:44:16.941993 4003 ts_manager.cc:194] Registered new tserver with Master: 5c7223cd519f40bc8da8e4b71a323349 (127.0.97.2:44415)
I20260928 08:44:16.943006 4003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:58279
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:44:16.963845 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39119
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:17.307488 4331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:17.307798 4331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:17.308048 4331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:17.315400 4331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:17.315517 4331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:17.315631 4331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:44:17.325626 4331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:17.326871 4331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:17.328333 4331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:17.341364 4337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:17.341415 4336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:17.341395 4339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:17.342361 4331 server_base.cc:1034] running on GCE node
I20260928 08:44:17.343036 4331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:17.344201 4331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:17.345484 4331 hybrid_clock.cc:648] HybridClock initialized: now 1790585057345449 us; error 81 us; skew 500 ppm
I20260928 08:44:17.345842 4331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:17 dist-test-slave-n6xs krb5kdc[3958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:17.349092 4331 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:44:17.350615 4331 webserver.cc:460] Webserver started at http://127.0.97.3:43371/ using document root <none> and password file <none>
I20260928 08:44:17.351361 4331 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:17.351476 4331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:17.351755 4331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:17.354700 4331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-2/data/instance:
uuid: "ebf864272b4a43808a3c152e83b99e9b"
format_stamp: "Formatted at 2026-09-28 08:44:17 on dist-test-slave-n6xs"
server_key: "a503d9f3d0bed531e002e495ebb69e5a"
server_key_iv: "3619f0aca2000a585c693003697d2f13"
server_key_version: "encryptionkey@0"
I20260928 08:44:17.355425 4331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-2/wal/instance:
uuid: "ebf864272b4a43808a3c152e83b99e9b"
format_stamp: "Formatted at 2026-09-28 08:44:17 on dist-test-slave-n6xs"
server_key: "a503d9f3d0bed531e002e495ebb69e5a"
server_key_iv: "3619f0aca2000a585c693003697d2f13"
server_key_version: "encryptionkey@0"
I20260928 08:44:17.360683 4331 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260928 08:44:17.364149 4346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:17.365499 4331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 08:44:17.365669 4331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-2/wal
uuid: "ebf864272b4a43808a3c152e83b99e9b"
format_stamp: "Formatted at 2026-09-28 08:44:17 on dist-test-slave-n6xs"
server_key: "a503d9f3d0bed531e002e495ebb69e5a"
server_key_iv: "3619f0aca2000a585c693003697d2f13"
server_key_version: "encryptionkey@0"
I20260928 08:44:17.365854 4331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:17.388955 4331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:17.393558 4331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:17.393944 4331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:17.395032 4331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:44:17.396715 4331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:44:17.396792 4331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:17.396977 4331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:44:17.397042 4331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:17.438675 4331 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37585
I20260928 08:44:17.438724 4458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37585 every 8 connection(s)
I20260928 08:44:17.440155 4331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:44:17.444291 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 4331
I20260928 08:44:17.444466 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790585022442112-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:44:17.445015 388 external_mini_cluster.cc:1428] Setting key 8f29f3d9fa94ff1bca28cebfc19cb470
I20260928 08:44:17.447357 4192 heartbeater.cc:503] Master 127.0.97.62:39119 was elected leader, sending a full tablet report...
Sep 28 08:44:17 dist-test-slave-n6xs krb5kdc[3958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585057, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:44:17.461407 4461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:17.442804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:41411 (local address 127.0.97.62:39119)
0928 08:44:17.443284 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:17.443291 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:17.444732 (+ 1441us) server_negotiation.cc:407] Connection header received
0928 08:44:17.446852 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:17.446856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:17.446931 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:17.447088 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:17.449062 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:17.449882 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:17.451495 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:17.452028 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:17.452126 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:17.455523 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:17.455528 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:17.455531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:17.455560 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:17.458171 (+ 2611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:17.458856 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:17.458861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:17.458866 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:17.458986 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:17.459451 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:17.459459 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:17.459463 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:17.459755 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:17.459988 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:17.460822 (+ 834us) server_negotiation.cc:299] Negotiation successful
0928 08:44:17.461030 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":224,"threads_started":1}
I20260928 08:44:17.462423 4462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:17.443503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39119 (local address 127.0.97.3:41411)
0928 08:44:17.444530 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:17.444592 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:17.446498 (+ 1906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:17.447387 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:17.447401 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:17.448168 (+ 767us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:17.448859 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:17.448886 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:17.450023 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:17.450027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:17.451360 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:17.451371 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:17.452244 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:17.452249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:17.452468 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:17.453090 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:17.453113 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:17.455320 (+ 2207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:17.458297 (+ 2977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:17.458305 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:17.458318 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:17.458724 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:17.459080 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:17.459083 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:17.459085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:17.459343 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:17.459973 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:17.459981 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:17.460322 (+ 341us) client_negotiation.cc:769] Sending connection context
0928 08:44:17.460717 (+ 395us) client_negotiation.cc:240] Negotiation successful
0928 08:44:17.461272 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":393,"threads_started":1}
I20260928 08:44:17.463747 4459 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39119
I20260928 08:44:17.464145 4459 heartbeater.cc:464] Registering TS with master...
I20260928 08:44:17.464875 4459 heartbeater.cc:511] Master 127.0.97.62:39119 requested a full tablet report, sending...
I20260928 08:44:17.466259 4002 ts_manager.cc:194] Registered new tserver with Master: ebf864272b4a43808a3c152e83b99e9b (127.0.97.3:37585)
I20260928 08:44:17.467090 4002 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:41411
I20260928 08:44:17.474016 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 08:44:17 dist-test-slave-n6xs krb5kdc[3958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 08:44:17 dist-test-slave-n6xs krb5kdc[3958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:44:17.523289 4467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:17.507178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37042 (local address 127.0.97.1:37877)
0928 08:44:17.507764 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:17.507772 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:17.507866 (+ 94us) server_negotiation.cc:407] Connection header received
0928 08:44:17.507943 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:17.507950 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:17.508180 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:17.508361 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:17.509783 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:17.510850 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:17.512199 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:17.512786 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:17.512951 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:17.517085 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:17.517100 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:17.517116 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:17.517155 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:17.520455 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:17.521172 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:17.521180 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:17.521183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:17.521313 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:17.521687 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:17.521693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:17.521696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:17.522040 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:17.522358 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:17.522644 (+ 286us) server_negotiation.cc:299] Negotiation successful
0928 08:44:17.522919 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":286,"threads_started":1}
Sep 28 08:44:17 dist-test-slave-n6xs krb5kdc[3958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:44:17.541265 4461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:17.527576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38698 (local address 127.0.97.62:39119)
0928 08:44:17.527792 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:17.527800 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:17.527991 (+ 191us) server_negotiation.cc:407] Connection header received
0928 08:44:17.528125 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:17.528128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:17.528191 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:17.528272 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:17.529513 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:17.530228 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:17.531458 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:17.531905 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:17.531999 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:17.536118 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:17.536124 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:17.536127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:17.536190 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:17.538487 (+ 2297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:17.539150 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:17.539154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:17.539156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:17.539344 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:17.540017 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:17.540021 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:17.540023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:17.540350 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:17.540536 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:17.540863 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 08:44:17.541028 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260928 08:44:17.541997 4002 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38698
I20260928 08:44:17.543183 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 4064
I20260928 08:44:17.556607 4467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:17.544087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37058 (local address 127.0.97.1:37877)
0928 08:44:17.544321 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:17.544327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:17.544574 (+ 247us) server_negotiation.cc:407] Connection header received
0928 08:44:17.544718 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:17.544722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:17.544827 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:17.544907 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:17.546152 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:17.546984 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:17.548249 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:17.548699 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:17.548797 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:17.551102 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:17.551106 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:17.551110 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:17.551141 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:17.553864 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:17.554664 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:17.554668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:17.554671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:17.554788 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:17.555188 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:17.555191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:17.555193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:17.555438 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:17.555618 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:17.556039 (+ 421us) server_negotiation.cc:299] Negotiation successful
0928 08:44:17.556227 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 08:44:17.557363 4187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:17.641426 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 4064
I20260928 08:44:17.658111 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 4198
Sep 28 08:44:17 dist-test-slave-n6xs krb5kdc[3958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 08:44:17.675765 4472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:17.659138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46624 (local address 127.0.97.2:44415)
0928 08:44:17.659775 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:17.659787 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:17.659849 (+ 62us) server_negotiation.cc:407] Connection header received
0928 08:44:17.659922 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:17.659929 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:17.660151 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:17.660281 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:17.661682 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:17.662813 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:17.664257 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:17.664852 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:17.664960 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:17.669419 (+ 4459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:17.669444 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:17.669461 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:17.669499 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:17.672945 (+ 3446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:17.673677 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:17.673685 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:17.673687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:17.673837 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:17.674229 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:17.674233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:17.674235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:17.674626 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:17.674855 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:17.675179 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 08:44:17.675445 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":329,"threads_started":1}
I20260928 08:44:17.677711 4321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:17.760305 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 4198
I20260928 08:44:17.776571 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 4331
Sep 28 08:44:17 dist-test-slave-n6xs krb5kdc[3958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 08:44:17.793422 4474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:17.777730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40852 (local address 127.0.97.3:37585)
0928 08:44:17.778385 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:17.778391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:17.778422 (+ 31us) server_negotiation.cc:407] Connection header received
0928 08:44:17.778482 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:17.778489 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:17.778705 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:17.778867 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:17.780173 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:17.781238 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:17.782541 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:17.783099 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:17.783196 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:17.787566 (+ 4370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:17.787580 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:17.787594 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:17.787634 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:17.790533 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:17.791223 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:17.791230 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:17.791232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:17.791348 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:17.791703 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:17.791709 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:17.791713 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:17.792173 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:17.792489 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:17.792798 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 08:44:17.793077 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":331,"threads_started":1}
I20260928 08:44:17.794754 4454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:17.878732 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 4331
I20260928 08:44:17.894605 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 3971
I20260928 08:44:17.896013 4033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:18.000494 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 3971
2026-09-28T08:44:18Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3897 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:44:18 dist-test-slave-n6xs krb5kdc[4479](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:44:18 dist-test-slave-n6xs krb5kdc[4479](info): set up 2 sockets
Sep 28 08:44:18 dist-test-slave-n6xs krb5kdc[4479](info): commencing operation
krb5kdc: starting...
W20260928 08:44:19.092315 388 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.003s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:44:19 dist-test-slave-n6xs krb5kdc[4479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:44:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:44:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:44:19.295149 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45979
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:19.614351 4492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:19.614624 4492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:19.614812 4492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:19.622411 4492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:44:19.622511 4492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:19.622617 4492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:19.622718 4492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:44:19.622804 4492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:44:19.632346 4492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:19.633605 4492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:19.634833 4492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:19.646369 4497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:19.646351 4500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:19.646318 4498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:19.647115 4492 server_base.cc:1034] running on GCE node
I20260928 08:44:19.647794 4492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:19.649170 4492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:19.650415 4492 hybrid_clock.cc:648] HybridClock initialized: now 1790585059650363 us; error 50 us; skew 500 ppm
I20260928 08:44:19.650768 4492 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:19 dist-test-slave-n6xs krb5kdc[4479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585059, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:19.653740 4492 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:44:19.654951 4492 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:19.655079 4492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:19.655427 4492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:19.658422 4492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "9810732d959649c7a221c8655b5d05f4"
format_stamp: "Formatted at 2026-09-28 08:44:19 on dist-test-slave-n6xs"
server_key: "a95724bc067ccc7163ef47bf51f3ce45"
server_key_iv: "517c1f48d0f805d76c6f4a44692a7cc1"
server_key_version: "encryptionkey@0"
I20260928 08:44:19.659176 4492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "9810732d959649c7a221c8655b5d05f4"
format_stamp: "Formatted at 2026-09-28 08:44:19 on dist-test-slave-n6xs"
server_key: "a95724bc067ccc7163ef47bf51f3ce45"
server_key_iv: "517c1f48d0f805d76c6f4a44692a7cc1"
server_key_version: "encryptionkey@0"
I20260928 08:44:19.664072 4492 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260928 08:44:19.667268 4506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:19.668612 4492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:44:19.668790 4492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "9810732d959649c7a221c8655b5d05f4"
format_stamp: "Formatted at 2026-09-28 08:44:19 on dist-test-slave-n6xs"
server_key: "a95724bc067ccc7163ef47bf51f3ce45"
server_key_iv: "517c1f48d0f805d76c6f4a44692a7cc1"
server_key_version: "encryptionkey@0"
I20260928 08:44:19.669003 4492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:19.684963 4492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:19.688786 4492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:19.689045 4492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:19.706847 4492 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45979
I20260928 08:44:19.706871 4557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45979 every 8 connection(s)
I20260928 08:44:19.707762 4492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:44:19.711346 4558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:19.714110 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 4492
I20260928 08:44:19.714308 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:44:19.715054 388 external_mini_cluster.cc:1428] Setting key 837d0e962c56e65b49c56d957bd9e46f
I20260928 08:44:19.723766 4558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9810732d959649c7a221c8655b5d05f4: Bootstrap starting.
I20260928 08:44:19.727555 4558 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9810732d959649c7a221c8655b5d05f4: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:19.728964 4558 log.cc:824] T 00000000000000000000000000000000 P 9810732d959649c7a221c8655b5d05f4: Log is configured to *not* fsync() on all Append() calls
Sep 28 08:44:19 dist-test-slave-n6xs krb5kdc[4479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:44:19.732197 4558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9810732d959649c7a221c8655b5d05f4: No bootstrap required, opened a new log
I20260928 08:44:19.737802 4561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:19.717124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58362 (local address 127.0.97.62:45979)
0928 08:44:19.717948 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:19.717958 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:19.717995 (+ 37us) server_negotiation.cc:407] Connection header received
0928 08:44:19.719524 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:19.719546 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:19.719999 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:19.720333 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:19.722113 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:19.723427 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:19.725154 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:19.725831 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:19.725957 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:19.730797 (+ 4840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:19.730805 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:19.730819 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:19.730855 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:19.733765 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:19.734556 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:19.734562 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:19.734569 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:19.734748 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:19.735122 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:19.735125 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:19.735127 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:19.735649 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:19.735962 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:19.736374 (+ 412us) server_negotiation.cc:299] Negotiation successful
0928 08:44:19.736727 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":387,"threads_started":1}
I20260928 08:44:19.741411 4558 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9810732d959649c7a221c8655b5d05f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9810732d959649c7a221c8655b5d05f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45979 } }
I20260928 08:44:19.741765 4558 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9810732d959649c7a221c8655b5d05f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:19.741870 4558 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9810732d959649c7a221c8655b5d05f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9810732d959649c7a221c8655b5d05f4, State: Initialized, Role: FOLLOWER
I20260928 08:44:19.742394 4558 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9810732d959649c7a221c8655b5d05f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9810732d959649c7a221c8655b5d05f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45979 } }
I20260928 08:44:19.742642 4558 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9810732d959649c7a221c8655b5d05f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:44:19.742782 4558 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9810732d959649c7a221c8655b5d05f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:44:19.742960 4558 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9810732d959649c7a221c8655b5d05f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:19.745136 4558 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9810732d959649c7a221c8655b5d05f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9810732d959649c7a221c8655b5d05f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45979 } }
I20260928 08:44:19.745620 4558 leader_election.cc:302] T 00000000000000000000000000000000 P 9810732d959649c7a221c8655b5d05f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9810732d959649c7a221c8655b5d05f4; no voters:
I20260928 08:44:19.746148 4558 leader_election.cc:288] T 00000000000000000000000000000000 P 9810732d959649c7a221c8655b5d05f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:44:19.746285 4563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9810732d959649c7a221c8655b5d05f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:44:19.747439 4563 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9810732d959649c7a221c8655b5d05f4 [term 1 LEADER]: Becoming Leader. State: Replica: 9810732d959649c7a221c8655b5d05f4, State: Running, Role: LEADER
I20260928 08:44:19.747987 4563 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9810732d959649c7a221c8655b5d05f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9810732d959649c7a221c8655b5d05f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45979 } }
I20260928 08:44:19.748194 4558 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9810732d959649c7a221c8655b5d05f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:44:19.751369 4564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9810732d959649c7a221c8655b5d05f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9810732d959649c7a221c8655b5d05f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9810732d959649c7a221c8655b5d05f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45979 } } }
I20260928 08:44:19.751340 4565 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9810732d959649c7a221c8655b5d05f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9810732d959649c7a221c8655b5d05f4. Latest consensus state: current_term: 1 leader_uuid: "9810732d959649c7a221c8655b5d05f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9810732d959649c7a221c8655b5d05f4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45979 } } }
I20260928 08:44:19.751744 4564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9810732d959649c7a221c8655b5d05f4 [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:19.751749 4565 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9810732d959649c7a221c8655b5d05f4 [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:19.754778 4573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:44:19.758867 4573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:44:19.766664 4573 catalog_manager.cc:1347] Generated new cluster ID: 50983bf4aa954b31afa412c3c998fd7e
I20260928 08:44:19.766847 4573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:44:19.777789 4573 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:44:19.779651 4573 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:44:19.789717 4573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9810732d959649c7a221c8655b5d05f4: Generated new TSK 0
I20260928 08:44:19.790736 4573 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:44:19.839730 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45979
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:20.175654 4584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:20.176084 4584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:20.176232 4584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:20.183449 4584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:20.183612 4584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:20.183796 4584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:44:20.193684 4584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:20.195029 4584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:20.196491 4584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:20.209863 4589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:20.209926 4590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:20.210098 4592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:20.210839 4584 server_base.cc:1034] running on GCE node
I20260928 08:44:20.211401 4584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:20.212353 4584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:20.213578 4584 hybrid_clock.cc:648] HybridClock initialized: now 1790585060213534 us; error 39 us; skew 500 ppm
I20260928 08:44:20.213881 4584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:20 dist-test-slave-n6xs krb5kdc[4479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:20.216956 4584 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:44:20.218022 4584 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:20.218103 4584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:20.218384 4584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:20.221252 4584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-0/data/instance:
uuid: "9c41ef4dec1240b3bb4951457623de9d"
format_stamp: "Formatted at 2026-09-28 08:44:20 on dist-test-slave-n6xs"
server_key: "741268362df77d5b6780991f81942eda"
server_key_iv: "23b89c5bd7ee8992f84109869c8b6466"
server_key_version: "encryptionkey@0"
I20260928 08:44:20.222005 4584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-0/wal/instance:
uuid: "9c41ef4dec1240b3bb4951457623de9d"
format_stamp: "Formatted at 2026-09-28 08:44:20 on dist-test-slave-n6xs"
server_key: "741268362df77d5b6780991f81942eda"
server_key_iv: "23b89c5bd7ee8992f84109869c8b6466"
server_key_version: "encryptionkey@0"
I20260928 08:44:20.226580 4584 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 08:44:20.229873 4598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:20.231307 4584 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:44:20.231487 4584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-0/wal
uuid: "9c41ef4dec1240b3bb4951457623de9d"
format_stamp: "Formatted at 2026-09-28 08:44:20 on dist-test-slave-n6xs"
server_key: "741268362df77d5b6780991f81942eda"
server_key_iv: "23b89c5bd7ee8992f84109869c8b6466"
server_key_version: "encryptionkey@0"
I20260928 08:44:20.231716 4584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:20.247871 4584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:20.252574 4584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:20.252852 4584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:20.253800 4584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:44:20.255281 4584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:44:20.255406 4584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:20.255535 4584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:44:20.255645 4584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:20.288424 4584 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40671
I20260928 08:44:20.288463 4710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40671 every 8 connection(s)
I20260928 08:44:20.289422 4584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:44:20.291733 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 4584
I20260928 08:44:20.291970 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:44:20.292896 388 external_mini_cluster.cc:1428] Setting key 5e38421c07dd57714daab335abbe04f0
Sep 28 08:44:20 dist-test-slave-n6xs krb5kdc[4479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:44:20.313112 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:20.293150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54947 (local address 127.0.97.62:45979)
0928 08:44:20.293610 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:20.293616 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:20.294772 (+ 1156us) server_negotiation.cc:407] Connection header received
0928 08:44:20.296455 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:20.296459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:20.296523 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:20.296620 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:20.298782 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:20.299546 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:20.301844 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:20.302256 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:20.302396 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:20.305878 (+ 3482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:20.305887 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:20.305895 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:20.305939 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:20.309509 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:20.310470 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:20.310474 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:20.310477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:20.310591 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:20.311182 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:20.311186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:20.311188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:20.311469 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:20.311611 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:20.312676 (+ 1065us) server_negotiation.cc:299] Negotiation successful
0928 08:44:20.312858 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":230,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 08:44:20.314579 4714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:20.292983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45979 (local address 127.0.97.1:54947)
0928 08:44:20.293943 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:20.293992 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:20.296153 (+ 2161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:20.296913 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:20.296928 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:20.297769 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:20.298586 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:20.298614 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:20.299713 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:20.299721 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:20.301677 (+ 1956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:20.301688 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:20.302494 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:20.302501 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:20.302718 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:20.303427 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:20.303452 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:20.305648 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:20.309645 (+ 3997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:20.309652 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:20.309666 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:20.310333 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:20.310704 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:20.310712 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:20.310718 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:20.311017 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:20.311629 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:20.311637 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:20.312185 (+ 548us) client_negotiation.cc:769] Sending connection context
0928 08:44:20.312527 (+ 342us) client_negotiation.cc:240] Negotiation successful
0928 08:44:20.313142 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":464,"threads_started":1}
I20260928 08:44:20.316251 4711 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260928 08:44:20.316596 4711 heartbeater.cc:464] Registering TS with master...
I20260928 08:44:20.317294 4711 heartbeater.cc:511] Master 127.0.97.62:45979 requested a full tablet report, sending...
I20260928 08:44:20.319336 4523 ts_manager.cc:194] Registered new tserver with Master: 9c41ef4dec1240b3bb4951457623de9d (127.0.97.1:40671)
I20260928 08:44:20.320828 4523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:54947
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 08:44:20.339120 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45979
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:20.665817 4717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:20.666082 4717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:20.666178 4717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:20.673454 4717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:20.673558 4717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:20.673755 4717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:44:20.683341 4717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:20.684620 4717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:20.685909 4717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:20.698556 4722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:20.698617 4725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:20.698593 4723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:20.701540 4717 server_base.cc:1034] running on GCE node
I20260928 08:44:20.702154 4717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:20.703109 4717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:20.704370 4717 hybrid_clock.cc:648] HybridClock initialized: now 1790585060704320 us; error 58 us; skew 500 ppm
I20260928 08:44:20.704713 4717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:20 dist-test-slave-n6xs krb5kdc[4479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:20.708025 4717 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:44:20.709211 4717 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:20.709339 4717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:20.709674 4717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:20.712531 4717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-1/data/instance:
uuid: "eb6bd7dffc1644499189489578d5e1c7"
format_stamp: "Formatted at 2026-09-28 08:44:20 on dist-test-slave-n6xs"
server_key: "2668b992130a50a8a1f92854c0e74257"
server_key_iv: "c37d446ba05d0153dd37bfd9288c47a0"
server_key_version: "encryptionkey@0"
I20260928 08:44:20.713315 4717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-1/wal/instance:
uuid: "eb6bd7dffc1644499189489578d5e1c7"
format_stamp: "Formatted at 2026-09-28 08:44:20 on dist-test-slave-n6xs"
server_key: "2668b992130a50a8a1f92854c0e74257"
server_key_iv: "c37d446ba05d0153dd37bfd9288c47a0"
server_key_version: "encryptionkey@0"
I20260928 08:44:20.718300 4717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:44:20.721769 4731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:20.723007 4717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:44:20.723194 4717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-1/wal
uuid: "eb6bd7dffc1644499189489578d5e1c7"
format_stamp: "Formatted at 2026-09-28 08:44:20 on dist-test-slave-n6xs"
server_key: "2668b992130a50a8a1f92854c0e74257"
server_key_iv: "c37d446ba05d0153dd37bfd9288c47a0"
server_key_version: "encryptionkey@0"
I20260928 08:44:20.723412 4717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:20.735738 4717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:20.739886 4717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:20.740157 4717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:20.741107 4717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:44:20.742331 4717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:44:20.742442 4717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:20.742589 4717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:44:20.742676 4717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:20.775992 4717 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35371
I20260928 08:44:20.776046 4843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35371 every 8 connection(s)
I20260928 08:44:20.776827 4717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:44:20.777892 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 4717
I20260928 08:44:20.778047 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:44:20.778584 388 external_mini_cluster.cc:1428] Setting key 0c4293b839207a828bd3027eeacd687d
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Sep 28 08:44:20 dist-test-slave-n6xs krb5kdc[4479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260928 08:44:20.803987 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:20.779621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51911 (local address 127.0.97.62:45979)
0928 08:44:20.779991 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:20.779997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:20.781098 (+ 1101us) server_negotiation.cc:407] Connection header received
0928 08:44:20.782883 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:20.782891 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:20.782985 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:20.783103 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:20.784675 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:20.785830 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:20.787373 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:20.788167 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:20.788318 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:20.797556 (+ 9238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:20.797561 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:20.797564 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:20.797596 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:20.800701 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:20.801476 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:20.801485 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:20.801491 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:20.801648 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:20.802051 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:20.802058 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:20.802064 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:20.802431 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:20.802588 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:20.803280 (+ 692us) server_negotiation.cc:299] Negotiation successful
0928 08:44:20.803525 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251}
I20260928 08:44:20.804936 4846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:20.780152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45979 (local address 127.0.97.2:51911)
0928 08:44:20.780935 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:20.780979 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:20.782625 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:20.783318 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:20.783326 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:20.783933 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:20.784496 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:20.784513 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:20.785981 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:20.785984 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:20.787186 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:20.787197 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:20.788363 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:20.788368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:20.788541 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:20.795042 (+ 6501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:20.795071 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:20.797266 (+ 2195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:20.800874 (+ 3608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:20.800880 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:20.800902 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:20.801335 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:20.801767 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:20.801770 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:20.801772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:20.801929 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:20.802568 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:20.802574 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:20.802884 (+ 310us) client_negotiation.cc:769] Sending connection context
0928 08:44:20.803138 (+ 254us) client_negotiation.cc:240] Negotiation successful
0928 08:44:20.803515 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":344,"threads_started":1}
I20260928 08:44:20.806514 4844 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45979
I20260928 08:44:20.806830 4844 heartbeater.cc:464] Registering TS with master...
I20260928 08:44:20.807616 4844 heartbeater.cc:511] Master 127.0.97.62:45979 requested a full tablet report, sending...
I20260928 08:44:20.809226 4523 ts_manager.cc:194] Registered new tserver with Master: eb6bd7dffc1644499189489578d5e1c7 (127.0.97.2:35371)
I20260928 08:44:20.810488 4523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:51911
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:44:20.827425 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45979
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:39197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:21.164335 4849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:21.164733 4849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:21.164942 4849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:21.172837 4849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:21.173013 4849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:21.173204 4849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:44:21.183234 4849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:39197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:21.184618 4849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:21.185967 4849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:21.198916 4854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:21.199060 4857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:21.198854 4855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:21.199697 4849 server_base.cc:1034] running on GCE node
I20260928 08:44:21.200448 4849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:21.201393 4849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:21.202627 4849 hybrid_clock.cc:648] HybridClock initialized: now 1790585061202557 us; error 63 us; skew 500 ppm
I20260928 08:44:21.202960 4849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:21 dist-test-slave-n6xs krb5kdc[4479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585061, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:21.206245 4849 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:44:21.207453 4849 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:21.207583 4849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:21.207980 4849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:21.210876 4849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-2/data/instance:
uuid: "19dfc2c4c2744e35a9d43549fe96b3a2"
format_stamp: "Formatted at 2026-09-28 08:44:21 on dist-test-slave-n6xs"
server_key: "067eb795d2577c3bbe2e30196b7cc365"
server_key_iv: "1929bd118ee560176b0759ea83698fa7"
server_key_version: "encryptionkey@0"
I20260928 08:44:21.211736 4849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-2/wal/instance:
uuid: "19dfc2c4c2744e35a9d43549fe96b3a2"
format_stamp: "Formatted at 2026-09-28 08:44:21 on dist-test-slave-n6xs"
server_key: "067eb795d2577c3bbe2e30196b7cc365"
server_key_iv: "1929bd118ee560176b0759ea83698fa7"
server_key_version: "encryptionkey@0"
I20260928 08:44:21.217005 4849 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 08:44:21.220633 4863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:21.221956 4849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:44:21.222162 4849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-2/wal
uuid: "19dfc2c4c2744e35a9d43549fe96b3a2"
format_stamp: "Formatted at 2026-09-28 08:44:21 on dist-test-slave-n6xs"
server_key: "067eb795d2577c3bbe2e30196b7cc365"
server_key_iv: "1929bd118ee560176b0759ea83698fa7"
server_key_version: "encryptionkey@0"
I20260928 08:44:21.222394 4849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:21.251598 4849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:21.255967 4849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:21.256259 4849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:21.257236 4849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:44:21.258517 4849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:44:21.258639 4849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:21.258777 4849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:44:21.258864 4849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:21.292209 4849 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45545
I20260928 08:44:21.292234 4975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45545 every 8 connection(s)
I20260928 08:44:21.293196 4849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:44:21.297366 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 4849
I20260928 08:44:21.297544 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:44:21.298017 388 external_mini_cluster.cc:1428] Setting key 2c549dbff87d561194041a334156e94f
Sep 28 08:44:21 dist-test-slave-n6xs krb5kdc[4479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585061, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:44:21.313784 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:21.296016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56049 (local address 127.0.97.62:45979)
0928 08:44:21.296243 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:21.296248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:21.297485 (+ 1237us) server_negotiation.cc:407] Connection header received
0928 08:44:21.299371 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:21.299376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:21.299459 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:21.299581 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:21.301281 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:21.302035 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:21.303588 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:21.304073 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:21.304207 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:21.307966 (+ 3759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:21.307972 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:21.307975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:21.308031 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:21.310584 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:21.311388 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:21.311392 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:21.311394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:21.311512 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:21.311998 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:21.312002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:21.312004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:21.312283 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:21.312452 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:21.313242 (+ 790us) server_negotiation.cc:299] Negotiation successful
0928 08:44:21.313459 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 08:44:21.314711 4978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:21.296526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45979 (local address 127.0.97.3:56049)
0928 08:44:21.297306 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:21.297350 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:21.299097 (+ 1747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:21.299876 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:21.299884 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:21.300406 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:21.301104 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:21.301122 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:21.302199 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:21.302203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:21.303445 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:21.303456 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:21.304306 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:21.304313 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:21.304533 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:21.305183 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:21.305259 (+ 76us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:21.307689 (+ 2430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:21.310748 (+ 3059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:21.310758 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:21.310795 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:21.311220 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:21.311624 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:21.311632 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:21.311635 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:21.311871 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:21.312429 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:21.312437 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:21.312790 (+ 353us) client_negotiation.cc:769] Sending connection context
0928 08:44:21.313072 (+ 282us) client_negotiation.cc:240] Negotiation successful
0928 08:44:21.313582 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":340,"threads_started":1}
I20260928 08:44:21.316196 4976 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45979
I20260928 08:44:21.316536 4976 heartbeater.cc:464] Registering TS with master...
I20260928 08:44:21.317178 4976 heartbeater.cc:511] Master 127.0.97.62:45979 requested a full tablet report, sending...
I20260928 08:44:21.318526 4523 ts_manager.cc:194] Registered new tserver with Master: 19dfc2c4c2744e35a9d43549fe96b3a2 (127.0.97.3:45545)
I20260928 08:44:21.319303 4523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:56049
I20260928 08:44:21.323977 4711 heartbeater.cc:503] Master 127.0.97.62:45979 was elected leader, sending a full tablet report...
I20260928 08:44:21.326092 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:44:21.342979 4713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:21.330786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58366 (local address 127.0.97.62:45979)
0928 08:44:21.330945 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:21.330950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:21.331507 (+ 557us) server_negotiation.cc:407] Connection header received
0928 08:44:21.331638 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:21.331642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:21.331715 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:21.331857 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:21.333274 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:21.333995 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:21.335216 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:21.335671 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:21.335864 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:21.338069 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:21.338074 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:21.338078 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:21.338108 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:21.340679 (+ 2571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:21.341251 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:21.341255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:21.341257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:21.341362 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:21.341742 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:21.341745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:21.341748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:21.342049 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:21.342304 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:21.342538 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 08:44:21.342702 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 08:44:21.347162 4522 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58366:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:44:21.354480 4522 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:44:21.379416 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:21.368356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48496 (local address 127.0.97.1:40671)
0928 08:44:21.369648 (+ 1292us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:21.369657 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:21.369681 (+ 24us) server_negotiation.cc:407] Connection header received
0928 08:44:21.369776 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:21.369782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:21.369965 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:21.370204 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:21.370876 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:21.372608 (+ 1732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:21.377073 (+ 4465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:21.378140 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:21.378220 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:21.378384 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:21.378910 (+ 526us) server_negotiation.cc:299] Negotiation successful
0928 08:44:21.379082 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1131,"thread_start_us":993,"threads_started":1}
I20260928 08:44:21.379858 4988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:21.367020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35371 (local address 127.0.0.1:39784)
0928 08:44:21.367893 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:21.367938 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:21.368135 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:21.371507 (+ 3372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:21.371513 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:21.371539 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:21.372034 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:21.372051 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:21.374255 (+ 2204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:21.374261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:21.376962 (+ 2701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:21.376979 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:21.378972 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:21.378979 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:21.379130 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:21.379161 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 08:44:21.379244 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 08:44:21.379435 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":22,"thread_start_us":382,"threads_started":1}
I20260928 08:44:21.380412 4990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:21.367741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40671 (local address 127.0.0.1:48496)
0928 08:44:21.368863 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:21.368890 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:21.369038 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:21.370193 (+ 1155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:21.370201 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:21.370230 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:21.370666 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:21.370680 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:21.372873 (+ 2193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:21.372879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:21.376875 (+ 3996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:21.376890 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:21.378507 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:21.378511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:21.378630 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:21.378665 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 08:44:21.379616 (+ 951us) client_negotiation.cc:240] Negotiation successful
0928 08:44:21.379761 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"mutex_wait_us":89,"thread_start_us":819,"threads_started":1}
I20260928 08:44:21.380447 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:21.367165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39784 (local address 127.0.97.2:35371)
0928 08:44:21.370837 (+ 3672us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:21.370847 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:21.370886 (+ 39us) server_negotiation.cc:407] Connection header received
0928 08:44:21.370986 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:21.370996 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:21.371260 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:21.372272 (+ 1012us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:21.372480 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:21.374066 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:21.377172 (+ 3106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:21.378674 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:21.379020 (+ 346us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:21.379251 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:21.379339 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 08:44:21.379903 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3504,"thread_start_us":3344,"threads_started":1}
I20260928 08:44:21.384400 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:21.369141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60356 (local address 127.0.97.3:45545)
0928 08:44:21.375429 (+ 6288us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:21.375440 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:21.376057 (+ 617us) server_negotiation.cc:407] Connection header received
0928 08:44:21.376145 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:21.376154 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:21.376370 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:21.376528 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:21.377451 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:21.379011 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:21.382096 (+ 3085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:21.383055 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:21.383171 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:21.383418 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:21.384038 (+ 620us) server_negotiation.cc:299] Negotiation successful
0928 08:44:21.384158 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6106,"thread_start_us":5931,"threads_started":1}
I20260928 08:44:21.384908 4992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:21.368474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45545 (local address 127.0.0.1:60356)
0928 08:44:21.375768 (+ 7294us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:21.375785 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:21.375946 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:21.376763 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:21.376767 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:21.376795 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:21.377260 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:21.377272 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:21.379934 (+ 2662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:21.379938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:21.381934 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:21.381945 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:21.383528 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:21.383534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:21.383681 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:21.383727 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 08:44:21.383825 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 08:44:21.384695 (+ 870us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7151,"thread_start_us":7014,"threads_started":1}
I20260928 08:44:21.384569 4646 tablet_service.cc:1467] Processing CreateTablet for tablet 4de56eb94fc24e8d8a33d9d28988724f (DEFAULT_TABLE table=test-table [id=1934114c1efe43889165a7b59ec7fc00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:21.384568 4779 tablet_service.cc:1467] Processing CreateTablet for tablet 4de56eb94fc24e8d8a33d9d28988724f (DEFAULT_TABLE table=test-table [id=1934114c1efe43889165a7b59ec7fc00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:21.386085 4646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4de56eb94fc24e8d8a33d9d28988724f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:21.386103 4779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4de56eb94fc24e8d8a33d9d28988724f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:21.388370 4911 tablet_service.cc:1467] Processing CreateTablet for tablet 4de56eb94fc24e8d8a33d9d28988724f (DEFAULT_TABLE table=test-table [id=1934114c1efe43889165a7b59ec7fc00]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:21.389544 4911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4de56eb94fc24e8d8a33d9d28988724f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:21.401741 4994 tablet_bootstrap.cc:492] T 4de56eb94fc24e8d8a33d9d28988724f P eb6bd7dffc1644499189489578d5e1c7: Bootstrap starting.
I20260928 08:44:21.403837 4996 tablet_bootstrap.cc:492] T 4de56eb94fc24e8d8a33d9d28988724f P 19dfc2c4c2744e35a9d43549fe96b3a2: Bootstrap starting.
I20260928 08:44:21.405834 4994 tablet_bootstrap.cc:654] T 4de56eb94fc24e8d8a33d9d28988724f P eb6bd7dffc1644499189489578d5e1c7: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:21.405483 4995 tablet_bootstrap.cc:492] T 4de56eb94fc24e8d8a33d9d28988724f P 9c41ef4dec1240b3bb4951457623de9d: Bootstrap starting.
I20260928 08:44:21.407972 4996 tablet_bootstrap.cc:654] T 4de56eb94fc24e8d8a33d9d28988724f P 19dfc2c4c2744e35a9d43549fe96b3a2: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:21.408025 4994 log.cc:824] T 4de56eb94fc24e8d8a33d9d28988724f P eb6bd7dffc1644499189489578d5e1c7: Log is configured to *not* fsync() on all Append() calls
I20260928 08:44:21.408737 4995 tablet_bootstrap.cc:654] T 4de56eb94fc24e8d8a33d9d28988724f P 9c41ef4dec1240b3bb4951457623de9d: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:21.409243 4996 log.cc:824] T 4de56eb94fc24e8d8a33d9d28988724f P 19dfc2c4c2744e35a9d43549fe96b3a2: Log is configured to *not* fsync() on all Append() calls
I20260928 08:44:21.409979 4995 log.cc:824] T 4de56eb94fc24e8d8a33d9d28988724f P 9c41ef4dec1240b3bb4951457623de9d: Log is configured to *not* fsync() on all Append() calls
I20260928 08:44:21.411226 4994 tablet_bootstrap.cc:492] T 4de56eb94fc24e8d8a33d9d28988724f P eb6bd7dffc1644499189489578d5e1c7: No bootstrap required, opened a new log
I20260928 08:44:21.411515 4994 ts_tablet_manager.cc:1397] T 4de56eb94fc24e8d8a33d9d28988724f P eb6bd7dffc1644499189489578d5e1c7: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 08:44:21.412252 4996 tablet_bootstrap.cc:492] T 4de56eb94fc24e8d8a33d9d28988724f P 19dfc2c4c2744e35a9d43549fe96b3a2: No bootstrap required, opened a new log
I20260928 08:44:21.412534 4996 ts_tablet_manager.cc:1397] T 4de56eb94fc24e8d8a33d9d28988724f P 19dfc2c4c2744e35a9d43549fe96b3a2: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.006s
I20260928 08:44:21.413483 4995 tablet_bootstrap.cc:492] T 4de56eb94fc24e8d8a33d9d28988724f P 9c41ef4dec1240b3bb4951457623de9d: No bootstrap required, opened a new log
I20260928 08:44:21.413702 4995 ts_tablet_manager.cc:1397] T 4de56eb94fc24e8d8a33d9d28988724f P 9c41ef4dec1240b3bb4951457623de9d: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260928 08:44:21.422879 4996 raft_consensus.cc:348] T 4de56eb94fc24e8d8a33d9d28988724f P 19dfc2c4c2744e35a9d43549fe96b3a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6bd7dffc1644499189489578d5e1c7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35371 } } peers { permanent_uuid: "19dfc2c4c2744e35a9d43549fe96b3a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45545 } } peers { permanent_uuid: "9c41ef4dec1240b3bb4951457623de9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40671 } }
I20260928 08:44:21.422915 4994 raft_consensus.cc:348] T 4de56eb94fc24e8d8a33d9d28988724f P eb6bd7dffc1644499189489578d5e1c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6bd7dffc1644499189489578d5e1c7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35371 } } peers { permanent_uuid: "19dfc2c4c2744e35a9d43549fe96b3a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45545 } } peers { permanent_uuid: "9c41ef4dec1240b3bb4951457623de9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40671 } }
I20260928 08:44:21.423321 4996 raft_consensus.cc:374] T 4de56eb94fc24e8d8a33d9d28988724f P 19dfc2c4c2744e35a9d43549fe96b3a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:21.423332 4994 raft_consensus.cc:374] T 4de56eb94fc24e8d8a33d9d28988724f P eb6bd7dffc1644499189489578d5e1c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:21.423421 4994 raft_consensus.cc:729] T 4de56eb94fc24e8d8a33d9d28988724f P eb6bd7dffc1644499189489578d5e1c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb6bd7dffc1644499189489578d5e1c7, State: Initialized, Role: FOLLOWER
I20260928 08:44:21.423471 4996 raft_consensus.cc:729] T 4de56eb94fc24e8d8a33d9d28988724f P 19dfc2c4c2744e35a9d43549fe96b3a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19dfc2c4c2744e35a9d43549fe96b3a2, State: Initialized, Role: FOLLOWER
I20260928 08:44:21.423928 4995 raft_consensus.cc:348] T 4de56eb94fc24e8d8a33d9d28988724f P 9c41ef4dec1240b3bb4951457623de9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6bd7dffc1644499189489578d5e1c7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35371 } } peers { permanent_uuid: "19dfc2c4c2744e35a9d43549fe96b3a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45545 } } peers { permanent_uuid: "9c41ef4dec1240b3bb4951457623de9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40671 } }
I20260928 08:44:21.424106 4994 consensus_queue.cc:260] T 4de56eb94fc24e8d8a33d9d28988724f P eb6bd7dffc1644499189489578d5e1c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6bd7dffc1644499189489578d5e1c7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35371 } } peers { permanent_uuid: "19dfc2c4c2744e35a9d43549fe96b3a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45545 } } peers { permanent_uuid: "9c41ef4dec1240b3bb4951457623de9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40671 } }
I20260928 08:44:21.424317 4995 raft_consensus.cc:374] T 4de56eb94fc24e8d8a33d9d28988724f P 9c41ef4dec1240b3bb4951457623de9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:21.424427 4995 raft_consensus.cc:729] T 4de56eb94fc24e8d8a33d9d28988724f P 9c41ef4dec1240b3bb4951457623de9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c41ef4dec1240b3bb4951457623de9d, State: Initialized, Role: FOLLOWER
I20260928 08:44:21.424299 4996 consensus_queue.cc:260] T 4de56eb94fc24e8d8a33d9d28988724f P 19dfc2c4c2744e35a9d43549fe96b3a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6bd7dffc1644499189489578d5e1c7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35371 } } peers { permanent_uuid: "19dfc2c4c2744e35a9d43549fe96b3a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45545 } } peers { permanent_uuid: "9c41ef4dec1240b3bb4951457623de9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40671 } }
I20260928 08:44:21.425024 4995 consensus_queue.cc:260] T 4de56eb94fc24e8d8a33d9d28988724f P 9c41ef4dec1240b3bb4951457623de9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6bd7dffc1644499189489578d5e1c7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35371 } } peers { permanent_uuid: "19dfc2c4c2744e35a9d43549fe96b3a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45545 } } peers { permanent_uuid: "9c41ef4dec1240b3bb4951457623de9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40671 } }
I20260928 08:44:21.425499 4976 heartbeater.cc:503] Master 127.0.97.62:45979 was elected leader, sending a full tablet report...
I20260928 08:44:21.426326 4996 ts_tablet_manager.cc:1428] T 4de56eb94fc24e8d8a33d9d28988724f P 19dfc2c4c2744e35a9d43549fe96b3a2: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260928 08:44:21.427464 4844 heartbeater.cc:503] Master 127.0.97.62:45979 was elected leader, sending a full tablet report...
I20260928 08:44:21.428269 4994 ts_tablet_manager.cc:1428] T 4de56eb94fc24e8d8a33d9d28988724f P eb6bd7dffc1644499189489578d5e1c7: Time spent starting tablet: real 0.017s user 0.006s sys 0.009s
I20260928 08:44:21.430367 4995 ts_tablet_manager.cc:1428] T 4de56eb94fc24e8d8a33d9d28988724f P 9c41ef4dec1240b3bb4951457623de9d: Time spent starting tablet: real 0.017s user 0.009s sys 0.008s
I20260928 08:44:21.454622 5000 raft_consensus.cc:482] T 4de56eb94fc24e8d8a33d9d28988724f P eb6bd7dffc1644499189489578d5e1c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:44:21.454924 5000 raft_consensus.cc:504] T 4de56eb94fc24e8d8a33d9d28988724f P eb6bd7dffc1644499189489578d5e1c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6bd7dffc1644499189489578d5e1c7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35371 } } peers { permanent_uuid: "19dfc2c4c2744e35a9d43549fe96b3a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45545 } } peers { permanent_uuid: "9c41ef4dec1240b3bb4951457623de9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40671 } }
I20260928 08:44:21.456768 5000 leader_election.cc:288] T 4de56eb94fc24e8d8a33d9d28988724f P eb6bd7dffc1644499189489578d5e1c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19dfc2c4c2744e35a9d43549fe96b3a2 (127.0.97.3:45545), 9c41ef4dec1240b3bb4951457623de9d (127.0.97.1:40671)
I20260928 08:44:21.463759 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:21.457221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44111 (local address 127.0.97.3:45545)
0928 08:44:21.457420 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:21.457426 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:21.458199 (+ 773us) server_negotiation.cc:407] Connection header received
0928 08:44:21.458349 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:21.458357 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:21.458434 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:21.458498 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:21.459203 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:21.460023 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:21.462108 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:21.462881 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:21.463024 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:21.463124 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:21.463391 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 08:44:21.463507 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 08:44:21.463966 5003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:21.456933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45545 (local address 127.0.97.2:44111)
0928 08:44:21.458022 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:21.458045 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:21.458216 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:21.458546 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:21.458551 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:21.458578 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:21.459037 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:21.459051 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:21.460152 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:21.460159 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:21.461963 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:21.461974 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:21.463081 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:21.463087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:21.463235 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:21.463258 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 08:44:21.463370 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 08:44:21.463498 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":909,"mutex_wait_us":21,"thread_start_us":573,"threads_started":1}
I20260928 08:44:21.464314 5004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:21.457244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40671 (local address 127.0.97.2:53525)
0928 08:44:21.458039 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:21.458053 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:21.458261 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:21.458665 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:21.458668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:21.458704 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:21.459089 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:21.459096 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:21.460443 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:21.460452 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:21.462164 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:21.462176 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:21.463352 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:21.463359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:21.463534 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:21.463555 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 08:44:21.463683 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 08:44:21.463831 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":34,"spinlock_wait_cycles":36736,"thread_start_us":542,"threads_started":1}
I20260928 08:44:21.464118 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:21.457332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53525 (local address 127.0.97.1:40671)
0928 08:44:21.457537 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:21.457543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:21.458143 (+ 600us) server_negotiation.cc:407] Connection header received
0928 08:44:21.458430 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:21.458435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:21.458534 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:21.458631 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:21.459280 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:21.460284 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:21.462326 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:21.463111 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:21.463288 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:21.463402 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:21.463710 (+ 308us) server_negotiation.cc:299] Negotiation successful
0928 08:44:21.463859 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 08:44:21.465121 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4de56eb94fc24e8d8a33d9d28988724f" candidate_uuid: "eb6bd7dffc1644499189489578d5e1c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19dfc2c4c2744e35a9d43549fe96b3a2" is_pre_election: true
I20260928 08:44:21.465548 4666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4de56eb94fc24e8d8a33d9d28988724f" candidate_uuid: "eb6bd7dffc1644499189489578d5e1c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c41ef4dec1240b3bb4951457623de9d" is_pre_election: true
I20260928 08:44:21.465692 4931 raft_consensus.cc:2454] T 4de56eb94fc24e8d8a33d9d28988724f P 19dfc2c4c2744e35a9d43549fe96b3a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6bd7dffc1644499189489578d5e1c7 in term 0.
I20260928 08:44:21.465986 4666 raft_consensus.cc:2454] T 4de56eb94fc24e8d8a33d9d28988724f P 9c41ef4dec1240b3bb4951457623de9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb6bd7dffc1644499189489578d5e1c7 in term 0.
I20260928 08:44:21.466516 4734 leader_election.cc:302] T 4de56eb94fc24e8d8a33d9d28988724f P eb6bd7dffc1644499189489578d5e1c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19dfc2c4c2744e35a9d43549fe96b3a2, eb6bd7dffc1644499189489578d5e1c7; no voters:
I20260928 08:44:21.466953 5000 raft_consensus.cc:2781] T 4de56eb94fc24e8d8a33d9d28988724f P eb6bd7dffc1644499189489578d5e1c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:44:21.467104 5000 raft_consensus.cc:482] T 4de56eb94fc24e8d8a33d9d28988724f P eb6bd7dffc1644499189489578d5e1c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:44:21.467273 5000 raft_consensus.cc:3037] T 4de56eb94fc24e8d8a33d9d28988724f P eb6bd7dffc1644499189489578d5e1c7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:21.469458 5000 raft_consensus.cc:504] T 4de56eb94fc24e8d8a33d9d28988724f P eb6bd7dffc1644499189489578d5e1c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6bd7dffc1644499189489578d5e1c7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35371 } } peers { permanent_uuid: "19dfc2c4c2744e35a9d43549fe96b3a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45545 } } peers { permanent_uuid: "9c41ef4dec1240b3bb4951457623de9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40671 } }
I20260928 08:44:21.470108 5000 leader_election.cc:288] T 4de56eb94fc24e8d8a33d9d28988724f P eb6bd7dffc1644499189489578d5e1c7 [CANDIDATE]: Term 1 election: Requested vote from peers 19dfc2c4c2744e35a9d43549fe96b3a2 (127.0.97.3:45545), 9c41ef4dec1240b3bb4951457623de9d (127.0.97.1:40671)
I20260928 08:44:21.470511 4931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4de56eb94fc24e8d8a33d9d28988724f" candidate_uuid: "eb6bd7dffc1644499189489578d5e1c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19dfc2c4c2744e35a9d43549fe96b3a2"
I20260928 08:44:21.470533 4666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4de56eb94fc24e8d8a33d9d28988724f" candidate_uuid: "eb6bd7dffc1644499189489578d5e1c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c41ef4dec1240b3bb4951457623de9d"
I20260928 08:44:21.470715 4931 raft_consensus.cc:3037] T 4de56eb94fc24e8d8a33d9d28988724f P 19dfc2c4c2744e35a9d43549fe96b3a2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:21.470717 4666 raft_consensus.cc:3037] T 4de56eb94fc24e8d8a33d9d28988724f P 9c41ef4dec1240b3bb4951457623de9d [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:21.472952 4666 raft_consensus.cc:2454] T 4de56eb94fc24e8d8a33d9d28988724f P 9c41ef4dec1240b3bb4951457623de9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6bd7dffc1644499189489578d5e1c7 in term 1.
I20260928 08:44:21.473049 4931 raft_consensus.cc:2454] T 4de56eb94fc24e8d8a33d9d28988724f P 19dfc2c4c2744e35a9d43549fe96b3a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb6bd7dffc1644499189489578d5e1c7 in term 1.
I20260928 08:44:21.473464 4735 leader_election.cc:302] T 4de56eb94fc24e8d8a33d9d28988724f P eb6bd7dffc1644499189489578d5e1c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c41ef4dec1240b3bb4951457623de9d, eb6bd7dffc1644499189489578d5e1c7; no voters:
I20260928 08:44:21.473791 5000 raft_consensus.cc:2781] T 4de56eb94fc24e8d8a33d9d28988724f P eb6bd7dffc1644499189489578d5e1c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:44:21.474303 5000 raft_consensus.cc:686] T 4de56eb94fc24e8d8a33d9d28988724f P eb6bd7dffc1644499189489578d5e1c7 [term 1 LEADER]: Becoming Leader. State: Replica: eb6bd7dffc1644499189489578d5e1c7, State: Running, Role: LEADER
I20260928 08:44:21.474821 5000 consensus_queue.cc:237] T 4de56eb94fc24e8d8a33d9d28988724f P eb6bd7dffc1644499189489578d5e1c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6bd7dffc1644499189489578d5e1c7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35371 } } peers { permanent_uuid: "19dfc2c4c2744e35a9d43549fe96b3a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45545 } } peers { permanent_uuid: "9c41ef4dec1240b3bb4951457623de9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40671 } }
I20260928 08:44:21.479336 4521 catalog_manager.cc:5495] T 4de56eb94fc24e8d8a33d9d28988724f P eb6bd7dffc1644499189489578d5e1c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb6bd7dffc1644499189489578d5e1c7 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "eb6bd7dffc1644499189489578d5e1c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb6bd7dffc1644499189489578d5e1c7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35371 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19dfc2c4c2744e35a9d43549fe96b3a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c41ef4dec1240b3bb4951457623de9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40671 } health_report { overall_health: UNKNOWN } } }
I20260928 08:44:21.517119 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:21.511680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39798 (local address 127.0.97.2:35371)
0928 08:44:21.511906 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:21.511911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:21.511927 (+ 16us) server_negotiation.cc:407] Connection header received
0928 08:44:21.512016 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:21.512020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:21.512098 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:21.512230 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:44:21.512606 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:21.513440 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:21.514811 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:21.515246 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:21.515354 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:21.515904 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:44:21.516452 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:44:21.516586 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:21.516706 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 08:44:21.516777 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 08:44:21.525570 4931 raft_consensus.cc:1261] T 4de56eb94fc24e8d8a33d9d28988724f P 19dfc2c4c2744e35a9d43549fe96b3a2 [term 1 FOLLOWER]: Refusing update from remote peer eb6bd7dffc1644499189489578d5e1c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:44:21.525571 4666 raft_consensus.cc:1261] T 4de56eb94fc24e8d8a33d9d28988724f P 9c41ef4dec1240b3bb4951457623de9d [term 1 FOLLOWER]: Refusing update from remote peer eb6bd7dffc1644499189489578d5e1c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:44:21.526700 5005 consensus_queue.cc:1035] T 4de56eb94fc24e8d8a33d9d28988724f P eb6bd7dffc1644499189489578d5e1c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c41ef4dec1240b3bb4951457623de9d" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:44:21.527087 5000 consensus_queue.cc:1035] T 4de56eb94fc24e8d8a33d9d28988724f P eb6bd7dffc1644499189489578d5e1c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19dfc2c4c2744e35a9d43549fe96b3a2" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 08:44:21.528625 4845 tablet.cc:2369] T 4de56eb94fc24e8d8a33d9d28988724f P eb6bd7dffc1644499189489578d5e1c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:44:21.555135 4521 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:44:21.555430 4521 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58366:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:44:21.559226 4521 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9810732d959649c7a221c8655b5d05f4: Sending DeleteTablet for 3 replicas of tablet 4de56eb94fc24e8d8a33d9d28988724f
I20260928 08:44:21.560557 4911 tablet_service.cc:1514] Processing DeleteTablet for tablet 4de56eb94fc24e8d8a33d9d28988724f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:60356
I20260928 08:44:21.560595 4646 tablet_service.cc:1514] Processing DeleteTablet for tablet 4de56eb94fc24e8d8a33d9d28988724f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48496
I20260928 08:44:21.561465 5018 tablet_replica.cc:330] T 4de56eb94fc24e8d8a33d9d28988724f P 9c41ef4dec1240b3bb4951457623de9d: stopping tablet replica
I20260928 08:44:21.561393 4779 tablet_service.cc:1514] Processing DeleteTablet for tablet 4de56eb94fc24e8d8a33d9d28988724f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:21 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:39784
I20260928 08:44:21.562029 5018 raft_consensus.cc:2229] T 4de56eb94fc24e8d8a33d9d28988724f P 9c41ef4dec1240b3bb4951457623de9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:44:21.562592 5018 raft_consensus.cc:2258] T 4de56eb94fc24e8d8a33d9d28988724f P 9c41ef4dec1240b3bb4951457623de9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:21.563354 5019 tablet_replica.cc:330] T 4de56eb94fc24e8d8a33d9d28988724f P 19dfc2c4c2744e35a9d43549fe96b3a2: stopping tablet replica
I20260928 08:44:21.563987 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 4584
I20260928 08:44:21.565339 5018 ts_tablet_manager.cc:1905] T 4de56eb94fc24e8d8a33d9d28988724f P 9c41ef4dec1240b3bb4951457623de9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:21.565475 5019 raft_consensus.cc:2229] T 4de56eb94fc24e8d8a33d9d28988724f P 19dfc2c4c2744e35a9d43549fe96b3a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:44:21.566375 5019 raft_consensus.cc:2258] T 4de56eb94fc24e8d8a33d9d28988724f P 19dfc2c4c2744e35a9d43549fe96b3a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:21.568670 5020 tablet_replica.cc:330] T 4de56eb94fc24e8d8a33d9d28988724f P eb6bd7dffc1644499189489578d5e1c7: stopping tablet replica
I20260928 08:44:21.569172 5020 raft_consensus.cc:2229] T 4de56eb94fc24e8d8a33d9d28988724f P eb6bd7dffc1644499189489578d5e1c7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:44:21.569568 5019 ts_tablet_manager.cc:1905] T 4de56eb94fc24e8d8a33d9d28988724f P 19dfc2c4c2744e35a9d43549fe96b3a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:21.570041 5020 raft_consensus.cc:2258] T 4de56eb94fc24e8d8a33d9d28988724f P eb6bd7dffc1644499189489578d5e1c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:21.572293 5020 ts_tablet_manager.cc:1905] T 4de56eb94fc24e8d8a33d9d28988724f P eb6bd7dffc1644499189489578d5e1c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:21.574657 5018 ts_tablet_manager.cc:1918] T 4de56eb94fc24e8d8a33d9d28988724f P 9c41ef4dec1240b3bb4951457623de9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:44:21.574892 5018 log.cc:1196] T 4de56eb94fc24e8d8a33d9d28988724f P 9c41ef4dec1240b3bb4951457623de9d: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-0/wal/wals/4de56eb94fc24e8d8a33d9d28988724f
I20260928 08:44:21.575528 5018 ts_tablet_manager.cc:1939] T 4de56eb94fc24e8d8a33d9d28988724f P 9c41ef4dec1240b3bb4951457623de9d: Deleting consensus metadata
I20260928 08:44:21.576512 5019 ts_tablet_manager.cc:1918] T 4de56eb94fc24e8d8a33d9d28988724f P 19dfc2c4c2744e35a9d43549fe96b3a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:44:21.576709 5019 log.cc:1196] T 4de56eb94fc24e8d8a33d9d28988724f P 19dfc2c4c2744e35a9d43549fe96b3a2: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-2/wal/wals/4de56eb94fc24e8d8a33d9d28988724f
I20260928 08:44:21.577278 5019 ts_tablet_manager.cc:1939] T 4de56eb94fc24e8d8a33d9d28988724f P 19dfc2c4c2744e35a9d43549fe96b3a2: Deleting consensus metadata
I20260928 08:44:21.577595 4510 catalog_manager.cc:4841] TS 9c41ef4dec1240b3bb4951457623de9d (127.0.97.1:40671): tablet 4de56eb94fc24e8d8a33d9d28988724f (table test-table [id=1934114c1efe43889165a7b59ec7fc00]) successfully deleted
I20260928 08:44:21.578959 4509 catalog_manager.cc:4841] TS 19dfc2c4c2744e35a9d43549fe96b3a2 (127.0.97.3:45545): tablet 4de56eb94fc24e8d8a33d9d28988724f (table test-table [id=1934114c1efe43889165a7b59ec7fc00]) successfully deleted
I20260928 08:44:21.581234 5020 ts_tablet_manager.cc:1918] T 4de56eb94fc24e8d8a33d9d28988724f P eb6bd7dffc1644499189489578d5e1c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:44:21.581441 5020 log.cc:1196] T 4de56eb94fc24e8d8a33d9d28988724f P eb6bd7dffc1644499189489578d5e1c7: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790585022442112-388-0/minicluster-data/ts-1/wal/wals/4de56eb94fc24e8d8a33d9d28988724f
I20260928 08:44:21.581996 5020 ts_tablet_manager.cc:1939] T 4de56eb94fc24e8d8a33d9d28988724f P eb6bd7dffc1644499189489578d5e1c7: Deleting consensus metadata
I20260928 08:44:21.583681 4509 catalog_manager.cc:4841] TS eb6bd7dffc1644499189489578d5e1c7 (127.0.97.2:35371): tablet 4de56eb94fc24e8d8a33d9d28988724f (table test-table [id=1934114c1efe43889165a7b59ec7fc00]) successfully deleted
Sep 28 08:44:21 dist-test-slave-n6xs krb5kdc[4479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:44:21.595400 4991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:21.565684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48502 (local address 127.0.97.1:40671)
0928 08:44:21.565863 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:21.565869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:21.568250 (+ 2381us) server_negotiation.cc:407] Connection header received
0928 08:44:21.578391 (+ 10141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:21.578399 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:21.578505 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:21.578587 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:21.580616 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:21.581762 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:21.583756 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:21.584435 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:21.584574 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:21.588846 (+ 4272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:21.588862 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:21.588881 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:21.588927 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:21.592645 (+ 3718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:21.593334 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:21.593344 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:21.593348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:21.593500 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:21.593873 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:21.593878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:21.593882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:21.594332 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:21.594635 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:21.594815 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 08:44:21.595042 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 08:44:21.596212 4706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:21.718854 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 4584
I20260928 08:44:21.741334 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 4717
Sep 28 08:44:21 dist-test-slave-n6xs krb5kdc[4479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 08:44:21.758405 4989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:21.742172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39806 (local address 127.0.97.2:35371)
0928 08:44:21.742353 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:21.742360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:21.742430 (+ 70us) server_negotiation.cc:407] Connection header received
0928 08:44:21.742609 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:21.742614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:21.742692 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:21.742814 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:21.744343 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:21.745541 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:21.746775 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:21.747409 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:21.747530 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:21.752057 (+ 4527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:21.752074 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:21.752096 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:21.752142 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:21.755384 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:21.756379 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:21.756387 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:21.756389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:21.756538 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:21.756925 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:21.756929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:21.756931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:21.757287 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:21.757565 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:21.757871 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 08:44:21.758038 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 08:44:21.759228 4839 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:21.870289 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 4717
I20260928 08:44:21.888360 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 4849
Sep 28 08:44:21 dist-test-slave-n6xs krb5kdc[4479](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585059, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 08:44:21.901996 4993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:21.889081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60364 (local address 127.0.97.3:45545)
0928 08:44:21.889241 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:21.889246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:21.889266 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:44:21.889424 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:21.889427 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:21.889498 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:21.889605 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:21.890780 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:21.891453 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:21.892674 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:21.893051 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:21.893141 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:21.897298 (+ 4157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:21.897310 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:21.897326 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:21.897361 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:21.899905 (+ 2544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:21.900455 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:21.900461 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:21.900463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:21.900578 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:21.900888 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:21.900891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:21.900893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:21.901188 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:21.901377 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:21.901598 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 08:44:21.901758 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 08:44:21.902616 4971 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:21.990654 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 4849
I20260928 08:44:22.006486 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 4492
I20260928 08:44:22.007519 4553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:22.115177 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 4492
2026-09-28T08:44:22Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4113 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T08:44:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:44:22Z Disabled control of system clock
I20260928 08:44:22.180065 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39347
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:45961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 08:44:22.498698 5029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:22.499024 5029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:22.499185 5029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:22.506595 5029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:44:22.506736 5029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:22.506832 5029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:22.506943 5029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:44:22.507030 5029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:44:22.516549 5029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39347
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:22.517824 5029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:22.519547 5029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:22.531757 5035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:22.531862 5037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:22.531795 5034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:22.532835 5029 server_base.cc:1034] running on GCE node
I20260928 08:44:22.533541 5029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:22.534829 5029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:22.536062 5029 hybrid_clock.cc:648] HybridClock initialized: now 1790585062536017 us; error 48 us; skew 500 ppm
I20260928 08:44:22.536428 5029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:44:22.538604 5029 webserver.cc:460] Webserver started at http://127.0.97.62:38429/ using document root <none> and password file <none>
I20260928 08:44:22.539260 5029 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:22.539382 5029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:22.539700 5029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:22.542573 5029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "c0182b213e424ab0a5aad6359208e5ee"
format_stamp: "Formatted at 2026-09-28 08:44:22 on dist-test-slave-n6xs"
server_key: "f379b246cde9b36c59c93ef806ab3d60"
server_key_iv: "6910027a567ea03cfdadc8a14a73e986"
server_key_version: "encryptionkey@0"
I20260928 08:44:22.543350 5029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "c0182b213e424ab0a5aad6359208e5ee"
format_stamp: "Formatted at 2026-09-28 08:44:22 on dist-test-slave-n6xs"
server_key: "f379b246cde9b36c59c93ef806ab3d60"
server_key_iv: "6910027a567ea03cfdadc8a14a73e986"
server_key_version: "encryptionkey@0"
I20260928 08:44:22.548409 5029 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 08:44:22.551648 5043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:22.552901 5029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:44:22.553066 5029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "c0182b213e424ab0a5aad6359208e5ee"
format_stamp: "Formatted at 2026-09-28 08:44:22 on dist-test-slave-n6xs"
server_key: "f379b246cde9b36c59c93ef806ab3d60"
server_key_iv: "6910027a567ea03cfdadc8a14a73e986"
server_key_version: "encryptionkey@0"
I20260928 08:44:22.553274 5029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:22.558934 5029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:22.559659 5029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:22.559988 5029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:22.578066 5029 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:39347
I20260928 08:44:22.578171 5094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:39347 every 8 connection(s)
I20260928 08:44:22.579396 5029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:44:22.583396 5095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:22.586238 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 5029
I20260928 08:44:22.586438 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:44:22.587082 388 external_mini_cluster.cc:1428] Setting key d953986ce7c3994673e314d22c81174a
I20260928 08:44:22.595325 5098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:22.589195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44064 (local address 127.0.97.62:39347)
0928 08:44:22.590176 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:22.590187 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:22.590227 (+ 40us) server_negotiation.cc:407] Connection header received
0928 08:44:22.591793 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:22.591862 (+ 69us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:22.592292 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:22.592619 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:22.593772 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:22.593778 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:22.593792 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:44:22.593806 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:22.593864 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:44:22.593881 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:22.594053 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:22.594172 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 08:44:22.594438 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":346,"threads_started":1}
I20260928 08:44:22.596274 5095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0182b213e424ab0a5aad6359208e5ee: Bootstrap starting.
I20260928 08:44:22.599534 5095 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0182b213e424ab0a5aad6359208e5ee: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:22.600581 5095 log.cc:824] T 00000000000000000000000000000000 P c0182b213e424ab0a5aad6359208e5ee: Log is configured to *not* fsync() on all Append() calls
I20260928 08:44:22.603346 5095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0182b213e424ab0a5aad6359208e5ee: No bootstrap required, opened a new log
I20260928 08:44:22.610860 5095 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0182b213e424ab0a5aad6359208e5ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0182b213e424ab0a5aad6359208e5ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39347 } }
I20260928 08:44:22.611177 5095 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0182b213e424ab0a5aad6359208e5ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:22.611258 5095 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0182b213e424ab0a5aad6359208e5ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0182b213e424ab0a5aad6359208e5ee, State: Initialized, Role: FOLLOWER
I20260928 08:44:22.611871 5095 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0182b213e424ab0a5aad6359208e5ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0182b213e424ab0a5aad6359208e5ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39347 } }
I20260928 08:44:22.612082 5095 raft_consensus.cc:388] T 00000000000000000000000000000000 P c0182b213e424ab0a5aad6359208e5ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:44:22.612190 5095 raft_consensus.cc:482] T 00000000000000000000000000000000 P c0182b213e424ab0a5aad6359208e5ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:44:22.612316 5095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0182b213e424ab0a5aad6359208e5ee [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:22.614425 5095 raft_consensus.cc:504] T 00000000000000000000000000000000 P c0182b213e424ab0a5aad6359208e5ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0182b213e424ab0a5aad6359208e5ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39347 } }
I20260928 08:44:22.614880 5095 leader_election.cc:302] T 00000000000000000000000000000000 P c0182b213e424ab0a5aad6359208e5ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c0182b213e424ab0a5aad6359208e5ee; no voters:
I20260928 08:44:22.615384 5095 leader_election.cc:288] T 00000000000000000000000000000000 P c0182b213e424ab0a5aad6359208e5ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:44:22.615482 5100 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c0182b213e424ab0a5aad6359208e5ee [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:44:22.616264 5100 raft_consensus.cc:686] T 00000000000000000000000000000000 P c0182b213e424ab0a5aad6359208e5ee [term 1 LEADER]: Becoming Leader. State: Replica: c0182b213e424ab0a5aad6359208e5ee, State: Running, Role: LEADER
I20260928 08:44:22.616824 5100 consensus_queue.cc:237] T 00000000000000000000000000000000 P c0182b213e424ab0a5aad6359208e5ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0182b213e424ab0a5aad6359208e5ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39347 } }
I20260928 08:44:22.617697 5095 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0182b213e424ab0a5aad6359208e5ee [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:44:22.619645 5102 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0182b213e424ab0a5aad6359208e5ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader c0182b213e424ab0a5aad6359208e5ee. Latest consensus state: current_term: 1 leader_uuid: "c0182b213e424ab0a5aad6359208e5ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0182b213e424ab0a5aad6359208e5ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39347 } } }
I20260928 08:44:22.620114 5102 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0182b213e424ab0a5aad6359208e5ee [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:22.621527 5101 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0182b213e424ab0a5aad6359208e5ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c0182b213e424ab0a5aad6359208e5ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c0182b213e424ab0a5aad6359208e5ee" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 39347 } } }
I20260928 08:44:22.622037 5101 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0182b213e424ab0a5aad6359208e5ee [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:22.622606 5108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:44:22.628223 5108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:44:22.636564 5108 catalog_manager.cc:1347] Generated new cluster ID: 7693e2a25bc7406b92148b91bc088486
I20260928 08:44:22.636672 5108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:44:22.663827 5108 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:44:22.664247 5108 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:44:22.672833 5108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c0182b213e424ab0a5aad6359208e5ee: Generated new TSK 0
I20260928 08:44:22.673738 5108 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:44:22.681945 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39347
--builtin_ntp_servers=127.0.97.20:45961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 08:44:23.003229 5119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:23.003489 5119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:23.003574 5119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:23.010794 5119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:23.010887 5119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:23.011044 5119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:44:23.020033 5119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:23.021459 5119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:23.023388 5119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:23.035709 5125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:23.035740 5127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:23.035663 5124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:23.036674 5119 server_base.cc:1034] running on GCE node
I20260928 08:44:23.037215 5119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:23.038121 5119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:23.039350 5119 hybrid_clock.cc:648] HybridClock initialized: now 1790585063039310 us; error 30 us; skew 500 ppm
I20260928 08:44:23.039719 5119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:44:23.042091 5119 webserver.cc:460] Webserver started at http://127.0.97.1:37083/ using document root <none> and password file <none>
I20260928 08:44:23.042832 5119 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:23.042977 5119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:23.043334 5119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:23.046401 5119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-0/data/instance:
uuid: "0822ed45b95d4e4693d7619cb93a7b27"
format_stamp: "Formatted at 2026-09-28 08:44:23 on dist-test-slave-n6xs"
server_key: "861a32bf721fdfe3f6456a218506c821"
server_key_iv: "efb9a09aab3033edf599da2d3cd0e168"
server_key_version: "encryptionkey@0"
I20260928 08:44:23.047250 5119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-0/wal/instance:
uuid: "0822ed45b95d4e4693d7619cb93a7b27"
format_stamp: "Formatted at 2026-09-28 08:44:23 on dist-test-slave-n6xs"
server_key: "861a32bf721fdfe3f6456a218506c821"
server_key_iv: "efb9a09aab3033edf599da2d3cd0e168"
server_key_version: "encryptionkey@0"
I20260928 08:44:23.052386 5119 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 08:44:23.055691 5133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:23.057173 5119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:44:23.057322 5119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-0/wal
uuid: "0822ed45b95d4e4693d7619cb93a7b27"
format_stamp: "Formatted at 2026-09-28 08:44:23 on dist-test-slave-n6xs"
server_key: "861a32bf721fdfe3f6456a218506c821"
server_key_iv: "efb9a09aab3033edf599da2d3cd0e168"
server_key_version: "encryptionkey@0"
I20260928 08:44:23.057533 5119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:23.063067 5119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:23.063802 5119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:23.064062 5119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:23.064877 5119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:44:23.066098 5119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:44:23.066179 5119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:23.066254 5119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:44:23.066308 5119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:23.096032 5119 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46575
I20260928 08:44:23.096117 5245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46575 every 8 connection(s)
I20260928 08:44:23.097334 5119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:44:23.102190 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 5119
I20260928 08:44:23.102385 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:44:23.102955 388 external_mini_cluster.cc:1428] Setting key ac3018955835f5c9dc6f400baf2ce20b
I20260928 08:44:23.105647 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39347
--builtin_ntp_servers=127.0.97.20:45961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 08:44:23.105859 5248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:23.099923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36739 (local address 127.0.97.62:39347)
0928 08:44:23.100376 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:23.100381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:23.101628 (+ 1247us) server_negotiation.cc:407] Connection header received
0928 08:44:23.103436 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:23.103442 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:23.103511 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:23.103618 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:23.105138 (+ 1520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:23.105143 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:23.105148 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:44:23.105165 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:23.105209 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:44:23.105220 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:23.105331 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:23.105521 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 08:44:23.105583 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":260,"threads_started":1}
I20260928 08:44:23.106771 5249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:23.100546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39347 (local address 127.0.97.1:36739)
0928 08:44:23.101464 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:23.101502 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:23.103176 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:23.103928 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:23.103939 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:23.104055 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:23.104798 (+ 743us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:44:23.104801 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:23.104850 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:44:23.104980 (+ 130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:23.105316 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:23.105326 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:23.105333 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 08:44:23.105601 (+ 268us) client_negotiation.cc:240] Negotiation successful
0928 08:44:23.105857 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":278,"threads_started":1}
I20260928 08:44:23.108281 5246 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39347
I20260928 08:44:23.108585 5246 heartbeater.cc:464] Registering TS with master...
I20260928 08:44:23.109248 5246 heartbeater.cc:511] Master 127.0.97.62:39347 requested a full tablet report, sending...
I20260928 08:44:23.111580 5060 ts_manager.cc:194] Registered new tserver with Master: 0822ed45b95d4e4693d7619cb93a7b27 (127.0.97.1:46575)
W20260928 08:44:23.422322 5250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:23.422643 5250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:23.422791 5250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:23.430025 5250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:23.430159 5250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:23.430336 5250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:44:23.439352 5250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:23.440672 5250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:23.442364 5250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:23.454809 5255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:23.454877 5256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:23.454851 5258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:23.455605 5250 server_base.cc:1034] running on GCE node
I20260928 08:44:23.456180 5250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:23.457064 5250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:23.458268 5250 hybrid_clock.cc:648] HybridClock initialized: now 1790585063458213 us; error 51 us; skew 500 ppm
I20260928 08:44:23.458559 5250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:44:23.460767 5250 webserver.cc:460] Webserver started at http://127.0.97.2:46805/ using document root <none> and password file <none>
I20260928 08:44:23.461472 5250 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:23.461596 5250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:23.461918 5250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:23.464665 5250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-1/data/instance:
uuid: "403a79bef200481f9bffc36dacbd48de"
format_stamp: "Formatted at 2026-09-28 08:44:23 on dist-test-slave-n6xs"
server_key: "838854608f1f1eeddb403f14dec741d2"
server_key_iv: "2b12b0bfb6245906db073cc1e3f4ce14"
server_key_version: "encryptionkey@0"
I20260928 08:44:23.465426 5250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-1/wal/instance:
uuid: "403a79bef200481f9bffc36dacbd48de"
format_stamp: "Formatted at 2026-09-28 08:44:23 on dist-test-slave-n6xs"
server_key: "838854608f1f1eeddb403f14dec741d2"
server_key_iv: "2b12b0bfb6245906db073cc1e3f4ce14"
server_key_version: "encryptionkey@0"
I20260928 08:44:23.470224 5250 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 08:44:23.473542 5264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:23.474858 5250 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:44:23.475028 5250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-1/wal
uuid: "403a79bef200481f9bffc36dacbd48de"
format_stamp: "Formatted at 2026-09-28 08:44:23 on dist-test-slave-n6xs"
server_key: "838854608f1f1eeddb403f14dec741d2"
server_key_iv: "2b12b0bfb6245906db073cc1e3f4ce14"
server_key_version: "encryptionkey@0"
I20260928 08:44:23.475242 5250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:23.480651 5250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:23.481379 5250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:23.481626 5250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:23.482472 5250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:44:23.483866 5250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:44:23.483973 5250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:23.484113 5250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:44:23.484196 5250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:23.516526 5250 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45035
I20260928 08:44:23.516571 5376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45035 every 8 connection(s)
I20260928 08:44:23.517886 5250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:44:23.524760 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 5250
I20260928 08:44:23.524968 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:44:23.525570 388 external_mini_cluster.cc:1428] Setting key a9a27e4aa53534c7f16a153ef4ed6bf8
I20260928 08:44:23.526726 5248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:23.520580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57117 (local address 127.0.97.62:39347)
0928 08:44:23.520757 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:23.520762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:23.522411 (+ 1649us) server_negotiation.cc:407] Connection header received
0928 08:44:23.524335 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:23.524339 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:23.524389 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:23.524473 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:23.525992 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:23.525996 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:23.525999 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:44:23.526027 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:23.526057 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:44:23.526068 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:23.526157 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:23.526386 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 08:44:23.526450 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 08:44:23.527882 5379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:23.521106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39347 (local address 127.0.97.2:57117)
0928 08:44:23.522269 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:23.522309 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:23.524034 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:23.524768 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:23.524781 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:23.524907 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:23.525637 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:44:23.525641 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:23.525693 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:44:23.525804 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:23.526167 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:23.526177 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:23.526184 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 08:44:23.526452 (+ 268us) client_negotiation.cc:240] Negotiation successful
0928 08:44:23.526719 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":383,"threads_started":1}
I20260928 08:44:23.528558 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39347
--builtin_ntp_servers=127.0.97.20:45961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 08:44:23.529281 5377 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39347
I20260928 08:44:23.529592 5377 heartbeater.cc:464] Registering TS with master...
I20260928 08:44:23.530233 5377 heartbeater.cc:511] Master 127.0.97.62:39347 requested a full tablet report, sending...
I20260928 08:44:23.531750 5060 ts_manager.cc:194] Registered new tserver with Master: 403a79bef200481f9bffc36dacbd48de (127.0.97.2:45035)
W20260928 08:44:23.854239 5380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:23.854494 5380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:23.854578 5380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:23.861891 5380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:23.861989 5380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:23.862102 5380 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:44:23.871003 5380 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:39347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:23.872292 5380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:23.873936 5380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:23.887005 5385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:23.886979 5386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:23.887073 5388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:23.888039 5380 server_base.cc:1034] running on GCE node
I20260928 08:44:23.888633 5380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:23.889575 5380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:23.890787 5380 hybrid_clock.cc:648] HybridClock initialized: now 1790585063890730 us; error 52 us; skew 500 ppm
I20260928 08:44:23.891152 5380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:44:23.893610 5380 webserver.cc:460] Webserver started at http://127.0.97.3:40209/ using document root <none> and password file <none>
I20260928 08:44:23.894318 5380 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:23.894438 5380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:23.894783 5380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:23.897876 5380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-2/data/instance:
uuid: "4d3fa56afc2840eeb86b86bb4c504333"
format_stamp: "Formatted at 2026-09-28 08:44:23 on dist-test-slave-n6xs"
server_key: "bf7c61a6ec2af89bf43ad91a01417b02"
server_key_iv: "a1ed439e76e90aa590135b309e167f96"
server_key_version: "encryptionkey@0"
I20260928 08:44:23.898705 5380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-2/wal/instance:
uuid: "4d3fa56afc2840eeb86b86bb4c504333"
format_stamp: "Formatted at 2026-09-28 08:44:23 on dist-test-slave-n6xs"
server_key: "bf7c61a6ec2af89bf43ad91a01417b02"
server_key_iv: "a1ed439e76e90aa590135b309e167f96"
server_key_version: "encryptionkey@0"
I20260928 08:44:23.903960 5380 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 08:44:23.907466 5394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:23.908780 5380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:44:23.908957 5380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-2/wal
uuid: "4d3fa56afc2840eeb86b86bb4c504333"
format_stamp: "Formatted at 2026-09-28 08:44:23 on dist-test-slave-n6xs"
server_key: "bf7c61a6ec2af89bf43ad91a01417b02"
server_key_iv: "a1ed439e76e90aa590135b309e167f96"
server_key_version: "encryptionkey@0"
I20260928 08:44:23.909193 5380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:23.914886 5380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:23.915668 5380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:23.916020 5380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:23.916915 5380 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:44:23.918401 5380 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:44:23.918524 5380 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:23.918660 5380 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:44:23.918756 5380 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:23.952075 5380 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42961
I20260928 08:44:23.952165 5506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42961 every 8 connection(s)
I20260928 08:44:23.953404 5380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:44:23.957985 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 5380
I20260928 08:44:23.958191 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:44:23.958593 388 external_mini_cluster.cc:1428] Setting key 95564b8cc600d2b1de10f3302b6b5128
I20260928 08:44:23.961853 5248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:23.955972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:53347 (local address 127.0.97.62:39347)
0928 08:44:23.956159 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:23.956163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:23.957743 (+ 1580us) server_negotiation.cc:407] Connection header received
0928 08:44:23.959460 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:23.959465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:23.959550 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:23.959656 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:23.960990 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:23.960996 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:23.961000 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:44:23.961019 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:23.961064 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:44:23.961079 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:23.961236 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:23.961434 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 08:44:23.961495 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 08:44:23.962709 5509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:23.956575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:39347 (local address 127.0.97.3:53347)
0928 08:44:23.957576 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:23.957634 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:23.959183 (+ 1549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:23.959924 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:23.959932 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:23.960024 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:23.960706 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:44:23.960708 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:23.960752 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:44:23.960835 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:23.961203 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:23.961210 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:23.961216 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 08:44:23.961481 (+ 265us) client_negotiation.cc:240] Negotiation successful
0928 08:44:23.961781 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":332,"threads_started":1}
I20260928 08:44:23.963965 5507 heartbeater.cc:347] Connected to a master server at 127.0.97.62:39347
I20260928 08:44:23.964217 5507 heartbeater.cc:464] Registering TS with master...
I20260928 08:44:23.964720 5507 heartbeater.cc:511] Master 127.0.97.62:39347 requested a full tablet report, sending...
I20260928 08:44:23.965896 5060 ts_manager.cc:194] Registered new tserver with Master: 4d3fa56afc2840eeb86b86bb4c504333 (127.0.97.3:42961)
I20260928 08:44:23.973879 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:44:23.980825 5248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:23.978318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38136 (local address 127.0.97.62:39347)
0928 08:44:23.978508 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:23.978516 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:23.978937 (+ 421us) server_negotiation.cc:407] Connection header received
0928 08:44:23.979189 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:23.979193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:23.979240 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:23.979333 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:23.980266 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:23.980270 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:23.980273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:44:23.980287 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:23.980316 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:44:23.980326 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:23.980417 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:23.980481 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 08:44:23.980521 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 08:44:23.984645 5060 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38136:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:44:23.991143 5060 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:44:24.022315 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:24.002902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42818 (local address 127.0.97.1:46575)
0928 08:44:24.019029 (+ 16127us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:24.019041 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:24.019076 (+ 35us) server_negotiation.cc:407] Connection header received
0928 08:44:24.019164 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:24.019175 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:24.019422 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:24.019589 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:24.021349 (+ 1760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:24.021357 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:24.021371 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:44:24.021396 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:24.021455 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:44:24.021477 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:24.021595 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:24.021884 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 08:44:24.021979 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15933,"thread_start_us":16977,"threads_started":1}
I20260928 08:44:24.023033 5519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:24.002414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46575 (local address 127.0.0.1:42818)
0928 08:44:24.018666 (+ 16252us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:24.018680 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:24.018797 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:24.019623 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:24.019630 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:24.019990 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:24.020638 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:44:24.020640 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:24.020674 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:44:24.020714 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:24.021658 (+ 944us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:24.021665 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:24.021671 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 08:44:24.021873 (+ 202us) client_negotiation.cc:240] Negotiation successful
0928 08:44:24.021949 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14933,"thread_start_us":9283,"threads_started":1}
I20260928 08:44:24.023921 5520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:24.002301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42961 (local address 127.0.0.1:46090)
0928 08:44:24.012512 (+ 10211us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:24.012559 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:24.012740 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:24.019173 (+ 6433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:24.019180 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:24.019270 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:24.020098 (+ 828us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:44:24.020104 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:24.020140 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:44:24.020246 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:24.023247 (+ 3001us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:24.023251 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:24.023259 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 08:44:24.023382 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 08:44:24.023432 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9891,"thread_start_us":15352,"threads_started":1}
I20260928 08:44:24.024055 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:24.003524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49030 (local address 127.0.97.2:45035)
0928 08:44:24.021043 (+ 17519us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:24.021053 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:24.021713 (+ 660us) server_negotiation.cc:407] Connection header received
0928 08:44:24.021802 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:24.021811 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:24.021988 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:24.022126 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:24.023105 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:24.023112 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:24.023123 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:44:24.023148 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:24.023216 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:44:24.023236 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:24.023383 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:24.023593 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 08:44:24.023686 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17347,"thread_start_us":17218,"threads_started":1}
I20260928 08:44:24.024107 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:24.002893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45035 (local address 127.0.0.1:49030)
0928 08:44:24.021473 (+ 18580us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:24.021494 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:24.021601 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:24.022146 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:24.022163 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:24.022183 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:24.022810 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:44:24.022815 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:24.022830 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:44:24.022875 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:24.023430 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:24.023435 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:24.023439 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 08:44:24.023581 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 08:44:24.023619 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18471,"thread_start_us":18339,"threads_started":1}
I20260928 08:44:24.025184 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:24.002506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46090 (local address 127.0.97.3:42961)
0928 08:44:24.018219 (+ 15713us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:24.018228 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:24.018264 (+ 36us) server_negotiation.cc:407] Connection header received
0928 08:44:24.018339 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:24.018347 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:24.018999 (+ 652us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:24.019154 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:24.023001 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:24.023008 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:24.023023 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:44:24.023041 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:24.023108 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:44:24.023126 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:24.024687 (+ 1561us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:24.024751 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 08:44:24.024831 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15548,"thread_start_us":15025,"threads_started":1}
I20260928 08:44:24.027278 5181 tablet_service.cc:1467] Processing CreateTablet for tablet 744d752a37994952be5393a6734f0bec (DEFAULT_TABLE table=test-table [id=1dd9dee7c94949c2852b3d54967de650]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:24.027781 5312 tablet_service.cc:1467] Processing CreateTablet for tablet 744d752a37994952be5393a6734f0bec (DEFAULT_TABLE table=test-table [id=1dd9dee7c94949c2852b3d54967de650]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:24.027912 5442 tablet_service.cc:1467] Processing CreateTablet for tablet 744d752a37994952be5393a6734f0bec (DEFAULT_TABLE table=test-table [id=1dd9dee7c94949c2852b3d54967de650]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:24.028827 5181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 744d752a37994952be5393a6734f0bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:24.029318 5442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 744d752a37994952be5393a6734f0bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:24.029572 5312 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 744d752a37994952be5393a6734f0bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:24.039248 5525 tablet_bootstrap.cc:492] T 744d752a37994952be5393a6734f0bec P 4d3fa56afc2840eeb86b86bb4c504333: Bootstrap starting.
I20260928 08:44:24.042994 5526 tablet_bootstrap.cc:492] T 744d752a37994952be5393a6734f0bec P 0822ed45b95d4e4693d7619cb93a7b27: Bootstrap starting.
I20260928 08:44:24.043059 5525 tablet_bootstrap.cc:654] T 744d752a37994952be5393a6734f0bec P 4d3fa56afc2840eeb86b86bb4c504333: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:24.044407 5525 log.cc:824] T 744d752a37994952be5393a6734f0bec P 4d3fa56afc2840eeb86b86bb4c504333: Log is configured to *not* fsync() on all Append() calls
I20260928 08:44:24.044622 5527 tablet_bootstrap.cc:492] T 744d752a37994952be5393a6734f0bec P 403a79bef200481f9bffc36dacbd48de: Bootstrap starting.
I20260928 08:44:24.046532 5526 tablet_bootstrap.cc:654] T 744d752a37994952be5393a6734f0bec P 0822ed45b95d4e4693d7619cb93a7b27: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:24.047550 5526 log.cc:824] T 744d752a37994952be5393a6734f0bec P 0822ed45b95d4e4693d7619cb93a7b27: Log is configured to *not* fsync() on all Append() calls
I20260928 08:44:24.047983 5525 tablet_bootstrap.cc:492] T 744d752a37994952be5393a6734f0bec P 4d3fa56afc2840eeb86b86bb4c504333: No bootstrap required, opened a new log
I20260928 08:44:24.047981 5527 tablet_bootstrap.cc:654] T 744d752a37994952be5393a6734f0bec P 403a79bef200481f9bffc36dacbd48de: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:24.048362 5525 ts_tablet_manager.cc:1397] T 744d752a37994952be5393a6734f0bec P 4d3fa56afc2840eeb86b86bb4c504333: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 08:44:24.049253 5527 log.cc:824] T 744d752a37994952be5393a6734f0bec P 403a79bef200481f9bffc36dacbd48de: Log is configured to *not* fsync() on all Append() calls
I20260928 08:44:24.051260 5526 tablet_bootstrap.cc:492] T 744d752a37994952be5393a6734f0bec P 0822ed45b95d4e4693d7619cb93a7b27: No bootstrap required, opened a new log
I20260928 08:44:24.051527 5526 ts_tablet_manager.cc:1397] T 744d752a37994952be5393a6734f0bec P 0822ed45b95d4e4693d7619cb93a7b27: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260928 08:44:24.052060 5527 tablet_bootstrap.cc:492] T 744d752a37994952be5393a6734f0bec P 403a79bef200481f9bffc36dacbd48de: No bootstrap required, opened a new log
I20260928 08:44:24.052316 5527 ts_tablet_manager.cc:1397] T 744d752a37994952be5393a6734f0bec P 403a79bef200481f9bffc36dacbd48de: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260928 08:44:24.058583 5525 raft_consensus.cc:348] T 744d752a37994952be5393a6734f0bec P 4d3fa56afc2840eeb86b86bb4c504333 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3fa56afc2840eeb86b86bb4c504333" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42961 } } peers { permanent_uuid: "0822ed45b95d4e4693d7619cb93a7b27" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46575 } } peers { permanent_uuid: "403a79bef200481f9bffc36dacbd48de" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45035 } }
I20260928 08:44:24.059031 5525 raft_consensus.cc:374] T 744d752a37994952be5393a6734f0bec P 4d3fa56afc2840eeb86b86bb4c504333 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:24.059139 5525 raft_consensus.cc:729] T 744d752a37994952be5393a6734f0bec P 4d3fa56afc2840eeb86b86bb4c504333 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d3fa56afc2840eeb86b86bb4c504333, State: Initialized, Role: FOLLOWER
I20260928 08:44:24.058944 5526 raft_consensus.cc:348] T 744d752a37994952be5393a6734f0bec P 0822ed45b95d4e4693d7619cb93a7b27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3fa56afc2840eeb86b86bb4c504333" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42961 } } peers { permanent_uuid: "0822ed45b95d4e4693d7619cb93a7b27" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46575 } } peers { permanent_uuid: "403a79bef200481f9bffc36dacbd48de" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45035 } }
I20260928 08:44:24.059407 5526 raft_consensus.cc:374] T 744d752a37994952be5393a6734f0bec P 0822ed45b95d4e4693d7619cb93a7b27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:24.059518 5526 raft_consensus.cc:729] T 744d752a37994952be5393a6734f0bec P 0822ed45b95d4e4693d7619cb93a7b27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0822ed45b95d4e4693d7619cb93a7b27, State: Initialized, Role: FOLLOWER
I20260928 08:44:24.059753 5525 consensus_queue.cc:260] T 744d752a37994952be5393a6734f0bec P 4d3fa56afc2840eeb86b86bb4c504333 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3fa56afc2840eeb86b86bb4c504333" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42961 } } peers { permanent_uuid: "0822ed45b95d4e4693d7619cb93a7b27" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46575 } } peers { permanent_uuid: "403a79bef200481f9bffc36dacbd48de" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45035 } }
I20260928 08:44:24.060148 5526 consensus_queue.cc:260] T 744d752a37994952be5393a6734f0bec P 0822ed45b95d4e4693d7619cb93a7b27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3fa56afc2840eeb86b86bb4c504333" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42961 } } peers { permanent_uuid: "0822ed45b95d4e4693d7619cb93a7b27" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46575 } } peers { permanent_uuid: "403a79bef200481f9bffc36dacbd48de" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45035 } }
I20260928 08:44:24.061018 5507 heartbeater.cc:503] Master 127.0.97.62:39347 was elected leader, sending a full tablet report...
I20260928 08:44:24.061510 5525 ts_tablet_manager.cc:1428] T 744d752a37994952be5393a6734f0bec P 4d3fa56afc2840eeb86b86bb4c504333: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260928 08:44:24.062741 5246 heartbeater.cc:503] Master 127.0.97.62:39347 was elected leader, sending a full tablet report...
I20260928 08:44:24.062865 5527 raft_consensus.cc:348] T 744d752a37994952be5393a6734f0bec P 403a79bef200481f9bffc36dacbd48de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3fa56afc2840eeb86b86bb4c504333" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42961 } } peers { permanent_uuid: "0822ed45b95d4e4693d7619cb93a7b27" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46575 } } peers { permanent_uuid: "403a79bef200481f9bffc36dacbd48de" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45035 } }
I20260928 08:44:24.063273 5527 raft_consensus.cc:374] T 744d752a37994952be5393a6734f0bec P 403a79bef200481f9bffc36dacbd48de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:24.063325 5526 ts_tablet_manager.cc:1428] T 744d752a37994952be5393a6734f0bec P 0822ed45b95d4e4693d7619cb93a7b27: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260928 08:44:24.063356 5527 raft_consensus.cc:729] T 744d752a37994952be5393a6734f0bec P 403a79bef200481f9bffc36dacbd48de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 403a79bef200481f9bffc36dacbd48de, State: Initialized, Role: FOLLOWER
I20260928 08:44:24.063916 5527 consensus_queue.cc:260] T 744d752a37994952be5393a6734f0bec P 403a79bef200481f9bffc36dacbd48de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3fa56afc2840eeb86b86bb4c504333" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42961 } } peers { permanent_uuid: "0822ed45b95d4e4693d7619cb93a7b27" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46575 } } peers { permanent_uuid: "403a79bef200481f9bffc36dacbd48de" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45035 } }
I20260928 08:44:24.064946 5377 heartbeater.cc:503] Master 127.0.97.62:39347 was elected leader, sending a full tablet report...
I20260928 08:44:24.066596 5527 ts_tablet_manager.cc:1428] T 744d752a37994952be5393a6734f0bec P 403a79bef200481f9bffc36dacbd48de: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
W20260928 08:44:24.099520 5247 tablet.cc:2369] T 744d752a37994952be5393a6734f0bec P 0822ed45b95d4e4693d7619cb93a7b27: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:44:24.128273 5533 raft_consensus.cc:482] T 744d752a37994952be5393a6734f0bec P 403a79bef200481f9bffc36dacbd48de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:44:24.128598 5533 raft_consensus.cc:504] T 744d752a37994952be5393a6734f0bec P 403a79bef200481f9bffc36dacbd48de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3fa56afc2840eeb86b86bb4c504333" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42961 } } peers { permanent_uuid: "0822ed45b95d4e4693d7619cb93a7b27" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46575 } } peers { permanent_uuid: "403a79bef200481f9bffc36dacbd48de" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45035 } }
I20260928 08:44:24.130225 5533 leader_election.cc:288] T 744d752a37994952be5393a6734f0bec P 403a79bef200481f9bffc36dacbd48de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d3fa56afc2840eeb86b86bb4c504333 (127.0.97.3:42961), 0822ed45b95d4e4693d7619cb93a7b27 (127.0.97.1:46575)
I20260928 08:44:24.133884 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:24.130641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59853 (local address 127.0.97.3:42961)
0928 08:44:24.130837 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:24.130843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:24.131656 (+ 813us) server_negotiation.cc:407] Connection header received
0928 08:44:24.131869 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:24.131875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:24.131935 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:24.132046 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:24.133254 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:24.133260 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:24.133265 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:44:24.133284 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:24.133322 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:44:24.133337 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:24.133460 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:24.133556 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 08:44:24.133613 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 08:44:24.133888 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:24.130711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59459 (local address 127.0.97.1:46575)
0928 08:44:24.130904 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:24.130912 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:24.131643 (+ 731us) server_negotiation.cc:407] Connection header received
0928 08:44:24.131869 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:24.131875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:24.131935 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:24.132046 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:24.133254 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:24.133260 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:24.133265 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:44:24.133284 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:24.133322 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:44:24.133337 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:24.133454 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:24.133557 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 08:44:24.133613 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 08:44:24.134181 5534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:24.130653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46575 (local address 127.0.97.2:59459)
0928 08:44:24.131505 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:24.131533 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:24.131665 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:24.132081 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:24.132086 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:24.132112 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:24.133045 (+ 933us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:44:24.133047 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:24.133058 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:44:24.133084 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:24.133439 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:24.133444 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:24.133449 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 08:44:24.133632 (+ 183us) client_negotiation.cc:240] Negotiation successful
0928 08:44:24.133691 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"mutex_wait_us":18,"thread_start_us":376,"threads_started":1}
I20260928 08:44:24.134327 5535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:24.130718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42961 (local address 127.0.97.2:59853)
0928 08:44:24.131537 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:24.131551 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:24.131698 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:24.132081 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:24.132093 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:24.132112 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:24.132991 (+ 879us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:44:24.132996 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:24.133019 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:44:24.133068 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:24.133439 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:24.133444 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:24.133449 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 08:44:24.133608 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 08:44:24.133680 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"spinlock_wait_cycles":22400,"thread_start_us":390,"threads_started":1}
I20260928 08:44:24.135224 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "744d752a37994952be5393a6734f0bec" candidate_uuid: "403a79bef200481f9bffc36dacbd48de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0822ed45b95d4e4693d7619cb93a7b27" is_pre_election: true
I20260928 08:44:24.135407 5462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "744d752a37994952be5393a6734f0bec" candidate_uuid: "403a79bef200481f9bffc36dacbd48de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3fa56afc2840eeb86b86bb4c504333" is_pre_election: true
I20260928 08:44:24.135735 5201 raft_consensus.cc:2454] T 744d752a37994952be5393a6734f0bec P 0822ed45b95d4e4693d7619cb93a7b27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 403a79bef200481f9bffc36dacbd48de in term 0.
I20260928 08:44:24.135984 5462 raft_consensus.cc:2454] T 744d752a37994952be5393a6734f0bec P 4d3fa56afc2840eeb86b86bb4c504333 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 403a79bef200481f9bffc36dacbd48de in term 0.
I20260928 08:44:24.136433 5266 leader_election.cc:302] T 744d752a37994952be5393a6734f0bec P 403a79bef200481f9bffc36dacbd48de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0822ed45b95d4e4693d7619cb93a7b27, 403a79bef200481f9bffc36dacbd48de; no voters:
I20260928 08:44:24.136791 5533 raft_consensus.cc:2781] T 744d752a37994952be5393a6734f0bec P 403a79bef200481f9bffc36dacbd48de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:44:24.136925 5533 raft_consensus.cc:482] T 744d752a37994952be5393a6734f0bec P 403a79bef200481f9bffc36dacbd48de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:44:24.137035 5533 raft_consensus.cc:3037] T 744d752a37994952be5393a6734f0bec P 403a79bef200481f9bffc36dacbd48de [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:24.139185 5533 raft_consensus.cc:504] T 744d752a37994952be5393a6734f0bec P 403a79bef200481f9bffc36dacbd48de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3fa56afc2840eeb86b86bb4c504333" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42961 } } peers { permanent_uuid: "0822ed45b95d4e4693d7619cb93a7b27" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46575 } } peers { permanent_uuid: "403a79bef200481f9bffc36dacbd48de" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45035 } }
I20260928 08:44:24.139722 5533 leader_election.cc:288] T 744d752a37994952be5393a6734f0bec P 403a79bef200481f9bffc36dacbd48de [CANDIDATE]: Term 1 election: Requested vote from peers 4d3fa56afc2840eeb86b86bb4c504333 (127.0.97.3:42961), 0822ed45b95d4e4693d7619cb93a7b27 (127.0.97.1:46575)
I20260928 08:44:24.140152 5462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "744d752a37994952be5393a6734f0bec" candidate_uuid: "403a79bef200481f9bffc36dacbd48de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d3fa56afc2840eeb86b86bb4c504333"
I20260928 08:44:24.140182 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "744d752a37994952be5393a6734f0bec" candidate_uuid: "403a79bef200481f9bffc36dacbd48de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0822ed45b95d4e4693d7619cb93a7b27"
I20260928 08:44:24.140343 5462 raft_consensus.cc:3037] T 744d752a37994952be5393a6734f0bec P 4d3fa56afc2840eeb86b86bb4c504333 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:24.140358 5201 raft_consensus.cc:3037] T 744d752a37994952be5393a6734f0bec P 0822ed45b95d4e4693d7619cb93a7b27 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:24.142455 5201 raft_consensus.cc:2454] T 744d752a37994952be5393a6734f0bec P 0822ed45b95d4e4693d7619cb93a7b27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 403a79bef200481f9bffc36dacbd48de in term 1.
I20260928 08:44:24.142475 5462 raft_consensus.cc:2454] T 744d752a37994952be5393a6734f0bec P 4d3fa56afc2840eeb86b86bb4c504333 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 403a79bef200481f9bffc36dacbd48de in term 1.
I20260928 08:44:24.142863 5266 leader_election.cc:302] T 744d752a37994952be5393a6734f0bec P 403a79bef200481f9bffc36dacbd48de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0822ed45b95d4e4693d7619cb93a7b27, 403a79bef200481f9bffc36dacbd48de; no voters:
I20260928 08:44:24.143131 5533 raft_consensus.cc:2781] T 744d752a37994952be5393a6734f0bec P 403a79bef200481f9bffc36dacbd48de [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:44:24.143550 5533 raft_consensus.cc:686] T 744d752a37994952be5393a6734f0bec P 403a79bef200481f9bffc36dacbd48de [term 1 LEADER]: Becoming Leader. State: Replica: 403a79bef200481f9bffc36dacbd48de, State: Running, Role: LEADER
I20260928 08:44:24.144066 5533 consensus_queue.cc:237] T 744d752a37994952be5393a6734f0bec P 403a79bef200481f9bffc36dacbd48de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3fa56afc2840eeb86b86bb4c504333" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42961 } } peers { permanent_uuid: "0822ed45b95d4e4693d7619cb93a7b27" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46575 } } peers { permanent_uuid: "403a79bef200481f9bffc36dacbd48de" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45035 } }
I20260928 08:44:24.148571 5058 catalog_manager.cc:5495] T 744d752a37994952be5393a6734f0bec P 403a79bef200481f9bffc36dacbd48de reported cstate change: term changed from 0 to 1, leader changed from <none> to 403a79bef200481f9bffc36dacbd48de (127.0.97.2). New cstate: current_term: 1 leader_uuid: "403a79bef200481f9bffc36dacbd48de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d3fa56afc2840eeb86b86bb4c504333" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0822ed45b95d4e4693d7619cb93a7b27" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "403a79bef200481f9bffc36dacbd48de" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45035 } health_report { overall_health: HEALTHY } } }
I20260928 08:44:24.184582 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:24.182669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49032 (local address 127.0.97.2:45035)
0928 08:44:24.182857 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:24.182862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:24.182879 (+ 17us) server_negotiation.cc:407] Connection header received
0928 08:44:24.183057 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:24.183061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:24.183171 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:24.183255 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:24.184031 (+ 776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:24.184035 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:24.184039 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:44:24.184052 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:24.184082 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:44:24.184092 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:24.184226 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:24.184279 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 08:44:24.184319 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 08:44:24.192658 5201 raft_consensus.cc:1261] T 744d752a37994952be5393a6734f0bec P 0822ed45b95d4e4693d7619cb93a7b27 [term 1 FOLLOWER]: Refusing update from remote peer 403a79bef200481f9bffc36dacbd48de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:44:24.192660 5462 raft_consensus.cc:1261] T 744d752a37994952be5393a6734f0bec P 4d3fa56afc2840eeb86b86bb4c504333 [term 1 FOLLOWER]: Refusing update from remote peer 403a79bef200481f9bffc36dacbd48de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:44:24.193701 5536 consensus_queue.cc:1035] T 744d752a37994952be5393a6734f0bec P 403a79bef200481f9bffc36dacbd48de [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d3fa56afc2840eeb86b86bb4c504333" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:44:24.194020 5533 consensus_queue.cc:1035] T 744d752a37994952be5393a6734f0bec P 403a79bef200481f9bffc36dacbd48de [LEADER]: Connected to new peer: Peer: permanent_uuid: "0822ed45b95d4e4693d7619cb93a7b27" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:44:24.204756 5539 mvcc.cc:205] Tried to move back new op lower bound from 7334236422922473472 to 7334236422738169856. Current Snapshot: MvccSnapshot[applied={T|T < 7334236422922473472}]
W20260928 08:44:24.205080 5508 tablet.cc:2369] T 744d752a37994952be5393a6734f0bec P 4d3fa56afc2840eeb86b86bb4c504333: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:44:24.205854 5540 mvcc.cc:205] Tried to move back new op lower bound from 7334236422922473472 to 7334236422738169856. Current Snapshot: MvccSnapshot[applied={T|T < 7334236422922473472}]
I20260928 08:44:24.218943 5058 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:44:24.219295 5058 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38136:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:44:24.222653 5058 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c0182b213e424ab0a5aad6359208e5ee: Sending DeleteTablet for 3 replicas of tablet 744d752a37994952be5393a6734f0bec
I20260928 08:44:24.223850 5442 tablet_service.cc:1514] Processing DeleteTablet for tablet 744d752a37994952be5393a6734f0bec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:24 UTC) from {username='slave'} at 127.0.0.1:46090
I20260928 08:44:24.223877 5181 tablet_service.cc:1514] Processing DeleteTablet for tablet 744d752a37994952be5393a6734f0bec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:24 UTC) from {username='slave'} at 127.0.0.1:42818
I20260928 08:44:24.224159 5312 tablet_service.cc:1514] Processing DeleteTablet for tablet 744d752a37994952be5393a6734f0bec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:24 UTC) from {username='slave'} at 127.0.0.1:49030
I20260928 08:44:24.226001 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 5119
I20260928 08:44:24.239187 5549 tablet_replica.cc:330] T 744d752a37994952be5393a6734f0bec P 4d3fa56afc2840eeb86b86bb4c504333: stopping tablet replica
I20260928 08:44:24.239703 5549 raft_consensus.cc:2229] T 744d752a37994952be5393a6734f0bec P 4d3fa56afc2840eeb86b86bb4c504333 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:44:24.240381 5549 raft_consensus.cc:2258] T 744d752a37994952be5393a6734f0bec P 4d3fa56afc2840eeb86b86bb4c504333 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:24.241976 5522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:24.227658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42828 (local address 127.0.97.1:46575)
0928 08:44:24.227918 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:24.227928 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:24.238843 (+ 10915us) server_negotiation.cc:407] Connection header received
0928 08:44:24.239317 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:24.239323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:24.239443 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:24.239522 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:24.241102 (+ 1580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:24.241109 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:24.241113 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:44:24.241134 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:24.241177 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:44:24.241193 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:24.241293 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:24.241564 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 08:44:24.241646 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 08:44:24.242918 5549 ts_tablet_manager.cc:1905] T 744d752a37994952be5393a6734f0bec P 4d3fa56afc2840eeb86b86bb4c504333: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:24.243178 5241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:24.251284 5550 tablet_replica.cc:330] T 744d752a37994952be5393a6734f0bec P 403a79bef200481f9bffc36dacbd48de: stopping tablet replica
I20260928 08:44:24.251856 5550 raft_consensus.cc:2229] T 744d752a37994952be5393a6734f0bec P 403a79bef200481f9bffc36dacbd48de [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:44:24.252682 5550 raft_consensus.cc:2258] T 744d752a37994952be5393a6734f0bec P 403a79bef200481f9bffc36dacbd48de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:24.255390 5550 ts_tablet_manager.cc:1905] T 744d752a37994952be5393a6734f0bec P 403a79bef200481f9bffc36dacbd48de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:24.265833 5549 ts_tablet_manager.cc:1918] T 744d752a37994952be5393a6734f0bec P 4d3fa56afc2840eeb86b86bb4c504333: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:44:24.266022 5549 log.cc:1196] T 744d752a37994952be5393a6734f0bec P 4d3fa56afc2840eeb86b86bb4c504333: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-2/wal/wals/744d752a37994952be5393a6734f0bec
I20260928 08:44:24.266551 5549 ts_tablet_manager.cc:1939] T 744d752a37994952be5393a6734f0bec P 4d3fa56afc2840eeb86b86bb4c504333: Deleting consensus metadata
I20260928 08:44:24.267252 5550 ts_tablet_manager.cc:1918] T 744d752a37994952be5393a6734f0bec P 403a79bef200481f9bffc36dacbd48de: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:44:24.267462 5550 log.cc:1196] T 744d752a37994952be5393a6734f0bec P 403a79bef200481f9bffc36dacbd48de: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790585022442112-388-0/minicluster-data/ts-1/wal/wals/744d752a37994952be5393a6734f0bec
I20260928 08:44:24.268188 5550 ts_tablet_manager.cc:1939] T 744d752a37994952be5393a6734f0bec P 403a79bef200481f9bffc36dacbd48de: Deleting consensus metadata
I20260928 08:44:24.268543 5044 catalog_manager.cc:4841] TS 4d3fa56afc2840eeb86b86bb4c504333 (127.0.97.3:42961): tablet 744d752a37994952be5393a6734f0bec (table test-table [id=1dd9dee7c94949c2852b3d54967de650]) successfully deleted
I20260928 08:44:24.270960 5046 catalog_manager.cc:4841] TS 403a79bef200481f9bffc36dacbd48de (127.0.97.2:45035): tablet 744d752a37994952be5393a6734f0bec (table test-table [id=1dd9dee7c94949c2852b3d54967de650]) successfully deleted
I20260928 08:44:24.383973 5548 tablet_replica.cc:330] T 744d752a37994952be5393a6734f0bec P 0822ed45b95d4e4693d7619cb93a7b27: stopping tablet replica
I20260928 08:44:24.384462 5548 raft_consensus.cc:2229] T 744d752a37994952be5393a6734f0bec P 0822ed45b95d4e4693d7619cb93a7b27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:44:24.384976 5548 raft_consensus.cc:2258] T 744d752a37994952be5393a6734f0bec P 0822ed45b95d4e4693d7619cb93a7b27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:24.385648 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 5119
W20260928 08:44:24.401922 5045 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:46575 (error 108)
I20260928 08:44:24.402669 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 5250
I20260928 08:44:24.404047 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:24.403500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46575 (local address 127.0.0.1:42834)
0928 08:44:24.403710 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:24.403875 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 08:44:24.404435 5045 catalog_manager.cc:4584] TS 0822ed45b95d4e4693d7619cb93a7b27 (127.0.97.1:46575): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 744d752a37994952be5393a6734f0bec: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46575: connect: Connection refused (error 111)
I20260928 08:44:24.405745 5524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:24.403747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49038 (local address 127.0.97.2:45035)
0928 08:44:24.403924 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:24.403932 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:24.403952 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:44:24.404042 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:24.404048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:24.404108 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:24.404216 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:24.405103 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:24.405109 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:24.405113 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:44:24.405136 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:24.405170 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:44:24.405182 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:24.405330 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:24.405388 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 08:44:24.405447 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 08:44:24.406430 5372 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:24.458156 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:24.457553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46575 (local address 127.0.0.1:42848)
0928 08:44:24.457783 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:24.457895 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 08:44:24.459215 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:24.458821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46575 (local address 127.0.0.1:42852)
0928 08:44:24.458968 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:24.459045 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 08:44:24.512729 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 5250
I20260928 08:44:24.519584 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:24.519116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46575 (local address 127.0.0.1:42864)
0928 08:44:24.519307 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:24.519406 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 08:44:24.520609 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:24.520293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46575 (local address 127.0.0.1:42870)
0928 08:44:24.520418 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:24.520489 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 08:44:24.530851 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 5380
I20260928 08:44:24.533363 5521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:24.531520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46104 (local address 127.0.97.3:42961)
0928 08:44:24.531681 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:24.531686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:24.531703 (+ 17us) server_negotiation.cc:407] Connection header received
0928 08:44:24.531919 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:24.531923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:24.532004 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:24.532117 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:24.532840 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:24.532845 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:24.532848 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:44:24.532869 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:24.532901 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:44:24.532912 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:24.533010 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:24.533121 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 08:44:24.533162 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 08:44:24.533952 5502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:24.600746 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:24.600295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46575 (local address 127.0.0.1:42872)
0928 08:44:24.600500 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:24.600610 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 08:44:24.601725 5523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:24.601338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46575 (local address 127.0.0.1:42876)
0928 08:44:24.601544 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:24.601618 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:46575: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260928 08:44:24.620631 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 5380
I20260928 08:44:24.637702 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 5029
I20260928 08:44:24.638852 5090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:24.743667 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 5029
2026-09-28T08:44:24Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2626 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 08:44:24.783571 388 mini_oidc.cc:152] Starting JWKS server
I20260928 08:44:24.786192 388 webserver.cc:460] Webserver started at https://127.0.0.1:40353/ using document root <none> and password file <none>
I20260928 08:44:24.786494 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 08:44:24.787125 388 webserver.cc:460] Webserver started at http://127.0.0.1:34905/ using document root <none> and password file <none>
2026-09-28T08:44:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:44:24Z Disabled control of system clock
I20260928 08:44:24.812184 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41625
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:40947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41625
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40353/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585022442112-388-0/testchainca.cert with env {}
W20260928 08:44:25.128928 5562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:25.129256 5562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:25.129416 5562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:25.136186 5562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:44:25.136319 5562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:25.136408 5562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:25.136503 5562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:44:25.136581 5562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 08:44:25.136670 5562 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 08:44:25.136722 5562 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40353/jwks.json
I20260928 08:44:25.145915 5562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41625
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40353/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585022442112-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:25.147120 5562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:25.148352 5562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:25.159777 5568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:25.159821 5570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:25.159844 5567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:25.160403 5562 server_base.cc:1034] running on GCE node
I20260928 08:44:25.161265 5562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:25.162501 5562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:25.163691 5562 hybrid_clock.cc:648] HybridClock initialized: now 1790585065163653 us; error 30 us; skew 500 ppm
I20260928 08:44:25.164155 5562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:44:25.166405 5562 webserver.cc:460] Webserver started at http://127.0.97.62:33037/ using document root <none> and password file <none>
I20260928 08:44:25.167057 5562 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:25.167173 5562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:25.167461 5562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:25.170214 5562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "93b9ee01adae4135a1b52ea76da1ebe7"
format_stamp: "Formatted at 2026-09-28 08:44:25 on dist-test-slave-n6xs"
server_key: "63b63ee19beffaeaabdab12eaff2dd2c"
server_key_iv: "b863d36f27cd70b2a4be79a08200846e"
server_key_version: "encryptionkey@0"
I20260928 08:44:25.170979 5562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "93b9ee01adae4135a1b52ea76da1ebe7"
format_stamp: "Formatted at 2026-09-28 08:44:25 on dist-test-slave-n6xs"
server_key: "63b63ee19beffaeaabdab12eaff2dd2c"
server_key_iv: "b863d36f27cd70b2a4be79a08200846e"
server_key_version: "encryptionkey@0"
I20260928 08:44:25.175645 5562 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260928 08:44:25.178797 5576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:25.180150 5562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:44:25.180320 5562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "93b9ee01adae4135a1b52ea76da1ebe7"
format_stamp: "Formatted at 2026-09-28 08:44:25 on dist-test-slave-n6xs"
server_key: "63b63ee19beffaeaabdab12eaff2dd2c"
server_key_iv: "b863d36f27cd70b2a4be79a08200846e"
server_key_version: "encryptionkey@0"
I20260928 08:44:25.180518 5562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:25.200260 5562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:25.201014 5562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:25.201295 5562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:25.218737 5562 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41625
I20260928 08:44:25.218756 5627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41625 every 8 connection(s)
I20260928 08:44:25.219998 5562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:44:25.223594 5628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:25.228461 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 5562
I20260928 08:44:25.228644 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:44:25.229421 388 external_mini_cluster.cc:1428] Setting key 499c14cbb1c5d0c081f09b0485d8f706
I20260928 08:44:25.235468 5628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93b9ee01adae4135a1b52ea76da1ebe7: Bootstrap starting.
I20260928 08:44:25.238878 5628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93b9ee01adae4135a1b52ea76da1ebe7: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:25.240060 5628 log.cc:824] T 00000000000000000000000000000000 P 93b9ee01adae4135a1b52ea76da1ebe7: Log is configured to *not* fsync() on all Append() calls
I20260928 08:44:25.243148 5628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93b9ee01adae4135a1b52ea76da1ebe7: No bootstrap required, opened a new log
I20260928 08:44:25.247294 5631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:25.231468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33128 (local address 127.0.97.62:41625)
0928 08:44:25.232647 (+ 1179us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:25.232661 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:25.236242 (+ 3581us) server_negotiation.cc:407] Connection header received
0928 08:44:25.238016 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:25.238035 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:25.238498 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:25.238796 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:25.239446 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:25.241151 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:25.243058 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:25.243748 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:25.243984 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:25.245178 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:25.245187 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:25.245200 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:44:25.245705 (+ 505us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:25.245786 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:44:25.245804 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:25.245898 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:25.246135 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 08:44:25.246438 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":682,"thread_start_us":401,"threads_started":1}
I20260928 08:44:25.250957 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:44:25.251236 5628 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93b9ee01adae4135a1b52ea76da1ebe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b9ee01adae4135a1b52ea76da1ebe7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41625 } }
I20260928 08:44:25.251602 5628 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93b9ee01adae4135a1b52ea76da1ebe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:25.251703 5628 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93b9ee01adae4135a1b52ea76da1ebe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93b9ee01adae4135a1b52ea76da1ebe7, State: Initialized, Role: FOLLOWER
I20260928 08:44:25.252229 5628 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93b9ee01adae4135a1b52ea76da1ebe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b9ee01adae4135a1b52ea76da1ebe7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41625 } }
I20260928 08:44:25.252440 5628 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93b9ee01adae4135a1b52ea76da1ebe7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:44:25.252561 5628 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93b9ee01adae4135a1b52ea76da1ebe7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:44:25.252724 5628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93b9ee01adae4135a1b52ea76da1ebe7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:25.255159 5628 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93b9ee01adae4135a1b52ea76da1ebe7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b9ee01adae4135a1b52ea76da1ebe7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41625 } }
I20260928 08:44:25.255607 5628 leader_election.cc:302] T 00000000000000000000000000000000 P 93b9ee01adae4135a1b52ea76da1ebe7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93b9ee01adae4135a1b52ea76da1ebe7; no voters:
I20260928 08:44:25.256222 5628 leader_election.cc:288] T 00000000000000000000000000000000 P 93b9ee01adae4135a1b52ea76da1ebe7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:44:25.256367 5633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93b9ee01adae4135a1b52ea76da1ebe7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:44:25.257406 5633 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93b9ee01adae4135a1b52ea76da1ebe7 [term 1 LEADER]: Becoming Leader. State: Replica: 93b9ee01adae4135a1b52ea76da1ebe7, State: Running, Role: LEADER
I20260928 08:44:25.257956 5633 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93b9ee01adae4135a1b52ea76da1ebe7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b9ee01adae4135a1b52ea76da1ebe7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41625 } }
I20260928 08:44:25.258255 5628 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93b9ee01adae4135a1b52ea76da1ebe7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:44:25.260479 5634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93b9ee01adae4135a1b52ea76da1ebe7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93b9ee01adae4135a1b52ea76da1ebe7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b9ee01adae4135a1b52ea76da1ebe7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41625 } } }
I20260928 08:44:25.260833 5634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93b9ee01adae4135a1b52ea76da1ebe7 [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:25.261286 5635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93b9ee01adae4135a1b52ea76da1ebe7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93b9ee01adae4135a1b52ea76da1ebe7. Latest consensus state: current_term: 1 leader_uuid: "93b9ee01adae4135a1b52ea76da1ebe7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93b9ee01adae4135a1b52ea76da1ebe7" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41625 } } }
I20260928 08:44:25.261571 5635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93b9ee01adae4135a1b52ea76da1ebe7 [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:25.266773 5639 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:44:25.271972 5639 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:44:25.279479 5639 catalog_manager.cc:1347] Generated new cluster ID: 8defd94fe58e4d7baf8a88f779aec2c5
I20260928 08:44:25.279579 5639 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:44:25.313489 5639 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:44:25.314769 5639 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:44:25.319793 5639 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93b9ee01adae4135a1b52ea76da1ebe7: Generated new TSK 0
I20260928 08:44:25.320798 5639 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 08:44:25.363488 5658 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 08:44:25.363663 5658 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 08:44:25.364550 5631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:25.343709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33140 (local address 127.0.97.62:41625)
0928 08:44:25.343974 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:25.343981 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:25.344395 (+ 414us) server_negotiation.cc:407] Connection header received
0928 08:44:25.344690 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:25.344693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:25.344755 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:25.344870 (+ 115us) server_negotiation.cc:226] Negotiated authn=JWT
0928 08:44:25.345456 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:25.346186 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:25.348250 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:25.348688 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:25.348804 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:25.349378 (+ 574us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 08:44:25.364075 (+ 14697us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40353/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 08:44:25.364359 (+ 284us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33140: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40353/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":78}
W20260928 08:44:25.364686 5631 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33140: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40353/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 08:44:25.366427 5657 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:44:25.343621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41625 (local address 127.0.0.1:33140)
0928 08:44:25.344288 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:25.344309 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:25.344496 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:25.344917 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:25.344924 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:25.344938 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0928 08:44:25.345309 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:25.345320 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:25.346330 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:25.346336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:25.348113 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:25.348128 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:25.348817 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:25.348880 (+ 63us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:25.349066 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:25.349139 (+ 73us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 08:44:25.364346 (+ 15207us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40353/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 08:44:25.364679 (+ 333us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:41625: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40353/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":323,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.97.62:41625: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40353/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 08:44:25.367905 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 5562
I20260928 08:44:25.369050 5623 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:25.464275 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 5562
2026-09-28T08:44:25Z chronyd exiting
I20260928 08:44:25.514286 388 test_util.cc:175] -----------------------------------------------
I20260928 08:44:25.514405 388 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790585022442112-388-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (735 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 08:44:25.517974 388 mini_oidc.cc:152] Starting JWKS server
I20260928 08:44:25.519644 388 webserver.cc:460] Webserver started at https://127.0.0.1:36889/ using document root <none> and password file <none>
I20260928 08:44:25.519902 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 08:44:25.520499 388 webserver.cc:460] Webserver started at http://127.0.0.1:36351/ using document root <none> and password file <none>
2026-09-28T08:44:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:44:25Z Disabled control of system clock
I20260928 08:44:25.545885 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37341
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:38403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37341
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36889/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585022442112-388-0/testchainca.cert with env {}
W20260928 08:44:25.861210 5665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:25.861487 5665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:25.861578 5665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:25.868913 5665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:44:25.869056 5665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:25.869136 5665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:25.869237 5665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:44:25.869313 5665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 08:44:25.869403 5665 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 08:44:25.869472 5665 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36889/jwks.json
I20260928 08:44:25.879508 5665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:38403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37341
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36889/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585022442112-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:25.880854 5665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:25.882149 5665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:25.894091 5671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:25.894100 5673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:25.894301 5670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:25.894919 5665 server_base.cc:1034] running on GCE node
I20260928 08:44:25.895704 5665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:25.897130 5665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:25.898350 5665 hybrid_clock.cc:648] HybridClock initialized: now 1790585065898299 us; error 45 us; skew 500 ppm
I20260928 08:44:25.898692 5665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:44:25.904663 5665 webserver.cc:460] Webserver started at http://127.0.97.62:43501/ using document root <none> and password file <none>
I20260928 08:44:25.905368 5665 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:25.905498 5665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:25.905818 5665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:25.908659 5665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "cdeacacf3ef54cb391884dac43796a1e"
format_stamp: "Formatted at 2026-09-28 08:44:25 on dist-test-slave-n6xs"
server_key: "cac2438bf444c56cb4c790e2ead04105"
server_key_iv: "db169eeb61e678082c4b40328ec4bf1f"
server_key_version: "encryptionkey@0"
I20260928 08:44:25.909427 5665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "cdeacacf3ef54cb391884dac43796a1e"
format_stamp: "Formatted at 2026-09-28 08:44:25 on dist-test-slave-n6xs"
server_key: "cac2438bf444c56cb4c790e2ead04105"
server_key_iv: "db169eeb61e678082c4b40328ec4bf1f"
server_key_version: "encryptionkey@0"
I20260928 08:44:25.914340 5665 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:44:25.917654 5679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:25.918941 5665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 08:44:25.919118 5665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "cdeacacf3ef54cb391884dac43796a1e"
format_stamp: "Formatted at 2026-09-28 08:44:25 on dist-test-slave-n6xs"
server_key: "cac2438bf444c56cb4c790e2ead04105"
server_key_iv: "db169eeb61e678082c4b40328ec4bf1f"
server_key_version: "encryptionkey@0"
I20260928 08:44:25.919320 5665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:25.935925 5665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:25.936714 5665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:25.936988 5665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:25.954759 5665 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37341
I20260928 08:44:25.954787 5730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37341 every 8 connection(s)
I20260928 08:44:25.956542 5665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:44:25.960659 5731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:25.962800 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 5665
I20260928 08:44:25.963012 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:44:25.963750 388 external_mini_cluster.cc:1428] Setting key e0e869a1de6eef469eedbac8c0fa6b2f
I20260928 08:44:25.974786 5731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdeacacf3ef54cb391884dac43796a1e: Bootstrap starting.
I20260928 08:44:25.978514 5731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdeacacf3ef54cb391884dac43796a1e: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:25.979884 5731 log.cc:824] T 00000000000000000000000000000000 P cdeacacf3ef54cb391884dac43796a1e: Log is configured to *not* fsync() on all Append() calls
I20260928 08:44:25.980690 5734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:25.965820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37342 (local address 127.0.97.62:37341)
0928 08:44:25.967379 (+ 1559us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:25.967400 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:25.968075 (+ 675us) server_negotiation.cc:407] Connection header received
0928 08:44:25.970330 (+ 2255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:25.970369 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:25.970992 (+ 623us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:25.971483 (+ 491us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:25.972060 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:25.973952 (+ 1892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:25.975416 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:25.976528 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:25.976730 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:25.977785 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:25.977795 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:25.977812 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:44:25.978509 (+ 697us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:25.978632 (+ 123us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:44:25.978658 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:25.978784 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:25.979032 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 08:44:25.979350 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":847,"thread_start_us":456,"threads_started":1}
I20260928 08:44:25.982952 5731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdeacacf3ef54cb391884dac43796a1e: No bootstrap required, opened a new log
I20260928 08:44:25.984464 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:44:25.990723 5731 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdeacacf3ef54cb391884dac43796a1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdeacacf3ef54cb391884dac43796a1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37341 } }
I20260928 08:44:25.991071 5731 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdeacacf3ef54cb391884dac43796a1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:25.991168 5731 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdeacacf3ef54cb391884dac43796a1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdeacacf3ef54cb391884dac43796a1e, State: Initialized, Role: FOLLOWER
I20260928 08:44:25.991716 5731 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdeacacf3ef54cb391884dac43796a1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdeacacf3ef54cb391884dac43796a1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37341 } }
I20260928 08:44:25.992007 5731 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdeacacf3ef54cb391884dac43796a1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:44:25.992148 5731 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdeacacf3ef54cb391884dac43796a1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:44:25.992301 5731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdeacacf3ef54cb391884dac43796a1e [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:25.994351 5731 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdeacacf3ef54cb391884dac43796a1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdeacacf3ef54cb391884dac43796a1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37341 } }
I20260928 08:44:25.994830 5731 leader_election.cc:302] T 00000000000000000000000000000000 P cdeacacf3ef54cb391884dac43796a1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdeacacf3ef54cb391884dac43796a1e; no voters:
I20260928 08:44:25.995347 5731 leader_election.cc:288] T 00000000000000000000000000000000 P cdeacacf3ef54cb391884dac43796a1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:44:25.995491 5736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdeacacf3ef54cb391884dac43796a1e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:44:25.996919 5736 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdeacacf3ef54cb391884dac43796a1e [term 1 LEADER]: Becoming Leader. State: Replica: cdeacacf3ef54cb391884dac43796a1e, State: Running, Role: LEADER
I20260928 08:44:25.997507 5731 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdeacacf3ef54cb391884dac43796a1e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:44:25.997399 5736 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdeacacf3ef54cb391884dac43796a1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdeacacf3ef54cb391884dac43796a1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37341 } }
I20260928 08:44:26.000311 5738 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdeacacf3ef54cb391884dac43796a1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdeacacf3ef54cb391884dac43796a1e. Latest consensus state: current_term: 1 leader_uuid: "cdeacacf3ef54cb391884dac43796a1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdeacacf3ef54cb391884dac43796a1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37341 } } }
I20260928 08:44:26.000411 5737 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdeacacf3ef54cb391884dac43796a1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdeacacf3ef54cb391884dac43796a1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdeacacf3ef54cb391884dac43796a1e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37341 } } }
I20260928 08:44:26.000723 5738 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdeacacf3ef54cb391884dac43796a1e [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:26.000789 5737 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdeacacf3ef54cb391884dac43796a1e [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:26.001706 5745 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:44:26.008137 5745 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:44:26.017697 5745 catalog_manager.cc:1347] Generated new cluster ID: f9100b7474fa41dc8ea0dbb1bc0e49f6
I20260928 08:44:26.017836 5745 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:44:26.044836 5745 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:44:26.046283 5745 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:44:26.054337 5745 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdeacacf3ef54cb391884dac43796a1e: Generated new TSK 0
I20260928 08:44:26.055182 5745 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 08:44:26.099612 5761 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 08:44:26.099748 5761 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 08:44:26.100977 5734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:26.073353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37344 (local address 127.0.97.62:37341)
0928 08:44:26.073510 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:26.073515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:26.074000 (+ 485us) server_negotiation.cc:407] Connection header received
0928 08:44:26.074226 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:26.074230 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:26.074291 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:26.074457 (+ 166us) server_negotiation.cc:226] Negotiated authn=JWT
0928 08:44:26.074934 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:26.075637 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:26.077024 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:26.077474 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:26.077599 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:26.078070 (+ 471us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 08:44:26.100235 (+ 22165us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36889/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 08:44:26.100654 (+ 419us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37344: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36889/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 08:44:26.101138 5734 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37344: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36889/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 08:44:26.101233 5760 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:44:26.073353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37341 (local address 127.0.0.1:37344)
0928 08:44:26.073900 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:26.073918 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:26.074068 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:26.074471 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:26.074476 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:26.074487 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0928 08:44:26.074752 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:26.074761 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:26.075762 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:26.075771 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:26.076868 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:26.076878 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:26.077601 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:26.077646 (+ 45us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:26.077800 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:26.077830 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 08:44:26.100498 (+ 22668us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36889/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 08:44:26.100799 (+ 301us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:37341: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36889/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":309,"threads_started":1}
I20260928 08:44:26.102563 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 5665
I20260928 08:44:26.103865 5726 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:26.198658 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 5665
2026-09-28T08:44:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (737 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 08:44:26.256036 388 mini_oidc.cc:152] Starting JWKS server
I20260928 08:44:26.257951 388 webserver.cc:460] Webserver started at https://127.0.0.1:46361/ using document root <none> and password file <none>
I20260928 08:44:26.258132 388 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 08:44:26.258795 388 webserver.cc:460] Webserver started at http://127.0.0.1:36791/ using document root <none> and password file <none>
2026-09-28T08:44:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:44:26Z Disabled control of system clock
I20260928 08:44:26.284688 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32791
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:46651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:32791
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46361/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 08:44:26.596894 5768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:26.597177 5768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:26.597260 5768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:26.604540 5768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:44:26.604650 5768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:26.604689 5768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:26.604746 5768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:44:26.604790 5768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 08:44:26.604843 5768 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 08:44:26.604882 5768 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46361/jwks.json
I20260928 08:44:26.613837 5768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:32791
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:32791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46361/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:26.615048 5768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:26.616778 5768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:26.628487 5774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:26.628444 5776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:26.628456 5773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:26.629325 5768 server_base.cc:1034] running on GCE node
I20260928 08:44:26.630080 5768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:26.631440 5768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:26.632661 5768 hybrid_clock.cc:648] HybridClock initialized: now 1790585066632620 us; error 36 us; skew 500 ppm
I20260928 08:44:26.632995 5768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:44:26.635120 5768 webserver.cc:460] Webserver started at http://127.0.97.62:45211/ using document root <none> and password file <none>
I20260928 08:44:26.635846 5768 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:26.635975 5768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:26.636288 5768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:26.639015 5768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "60553472730044ed9816d97cedfe34fc"
format_stamp: "Formatted at 2026-09-28 08:44:26 on dist-test-slave-n6xs"
server_key: "a58cca4392158cd65772efa7da79561e"
server_key_iv: "479a7b778fbbc7f143a36ddbf2f5caa6"
server_key_version: "encryptionkey@0"
I20260928 08:44:26.639773 5768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "60553472730044ed9816d97cedfe34fc"
format_stamp: "Formatted at 2026-09-28 08:44:26 on dist-test-slave-n6xs"
server_key: "a58cca4392158cd65772efa7da79561e"
server_key_iv: "479a7b778fbbc7f143a36ddbf2f5caa6"
server_key_version: "encryptionkey@0"
I20260928 08:44:26.644605 5768 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 08:44:26.647857 5782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:26.649123 5768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:44:26.649292 5768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "60553472730044ed9816d97cedfe34fc"
format_stamp: "Formatted at 2026-09-28 08:44:26 on dist-test-slave-n6xs"
server_key: "a58cca4392158cd65772efa7da79561e"
server_key_iv: "479a7b778fbbc7f143a36ddbf2f5caa6"
server_key_version: "encryptionkey@0"
I20260928 08:44:26.649482 5768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:26.662811 5768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:26.663498 5768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:26.663759 5768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:26.681427 5768 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:32791
I20260928 08:44:26.681453 5833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:32791 every 8 connection(s)
I20260928 08:44:26.682657 5768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:44:26.686400 5834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:26.690711 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 5768
I20260928 08:44:26.690902 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:44:26.691712 388 external_mini_cluster.cc:1428] Setting key 8fa6e069b83fa6fc7d58c58df0537c34
I20260928 08:44:26.697515 5834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60553472730044ed9816d97cedfe34fc: Bootstrap starting.
I20260928 08:44:26.700381 5834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60553472730044ed9816d97cedfe34fc: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:26.701345 5834 log.cc:824] T 00000000000000000000000000000000 P 60553472730044ed9816d97cedfe34fc: Log is configured to *not* fsync() on all Append() calls
I20260928 08:44:26.704119 5834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60553472730044ed9816d97cedfe34fc: No bootstrap required, opened a new log
I20260928 08:44:26.704468 5837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:26.693539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52164 (local address 127.0.97.62:32791)
0928 08:44:26.694425 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:26.694441 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:26.694484 (+ 43us) server_negotiation.cc:407] Connection header received
0928 08:44:26.696311 (+ 1827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:26.696331 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:26.696741 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:26.697111 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:26.697638 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:26.698844 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:26.700146 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:26.700869 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:26.701050 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:26.702213 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:26.702222 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:26.702241 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:44:26.702853 (+ 612us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:26.702921 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:44:26.702941 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:26.703037 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:26.703251 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 08:44:26.703539 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":332,"threads_started":1}
I20260928 08:44:26.708096 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:44:26.713343 5834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60553472730044ed9816d97cedfe34fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60553472730044ed9816d97cedfe34fc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32791 } }
I20260928 08:44:26.713639 5834 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60553472730044ed9816d97cedfe34fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:26.713711 5834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60553472730044ed9816d97cedfe34fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60553472730044ed9816d97cedfe34fc, State: Initialized, Role: FOLLOWER
I20260928 08:44:26.714227 5834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60553472730044ed9816d97cedfe34fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60553472730044ed9816d97cedfe34fc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32791 } }
I20260928 08:44:26.714437 5834 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60553472730044ed9816d97cedfe34fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:44:26.714535 5834 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60553472730044ed9816d97cedfe34fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:44:26.714651 5834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60553472730044ed9816d97cedfe34fc [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:26.716830 5834 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60553472730044ed9816d97cedfe34fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60553472730044ed9816d97cedfe34fc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32791 } }
I20260928 08:44:26.717242 5834 leader_election.cc:302] T 00000000000000000000000000000000 P 60553472730044ed9816d97cedfe34fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60553472730044ed9816d97cedfe34fc; no voters:
I20260928 08:44:26.717741 5834 leader_election.cc:288] T 00000000000000000000000000000000 P 60553472730044ed9816d97cedfe34fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:44:26.717895 5839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60553472730044ed9816d97cedfe34fc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:44:26.718577 5839 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60553472730044ed9816d97cedfe34fc [term 1 LEADER]: Becoming Leader. State: Replica: 60553472730044ed9816d97cedfe34fc, State: Running, Role: LEADER
I20260928 08:44:26.719120 5839 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60553472730044ed9816d97cedfe34fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60553472730044ed9816d97cedfe34fc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32791 } }
I20260928 08:44:26.719633 5834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60553472730044ed9816d97cedfe34fc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:44:26.722115 5840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60553472730044ed9816d97cedfe34fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60553472730044ed9816d97cedfe34fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60553472730044ed9816d97cedfe34fc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32791 } } }
I20260928 08:44:26.722076 5841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60553472730044ed9816d97cedfe34fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60553472730044ed9816d97cedfe34fc. Latest consensus state: current_term: 1 leader_uuid: "60553472730044ed9816d97cedfe34fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60553472730044ed9816d97cedfe34fc" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 32791 } } }
I20260928 08:44:26.722499 5841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60553472730044ed9816d97cedfe34fc [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:26.722499 5840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60553472730044ed9816d97cedfe34fc [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:26.723446 5846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:44:26.727599 5846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:44:26.736159 5846 catalog_manager.cc:1347] Generated new cluster ID: 56dd4b1a07194d5393bd4dd5211ed8cc
I20260928 08:44:26.736279 5846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:44:26.758176 5846 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:44:26.760005 5846 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:44:26.769366 5846 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60553472730044ed9816d97cedfe34fc: Generated new TSK 0
I20260928 08:44:26.770340 5846 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:44:26.799027 5837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:26.791407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52174 (local address 127.0.97.62:32791)
0928 08:44:26.791559 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:26.791566 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:26.792167 (+ 601us) server_negotiation.cc:407] Connection header received
0928 08:44:26.792408 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:26.792412 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:26.792482 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:26.792627 (+ 145us) server_negotiation.cc:226] Negotiated authn=JWT
0928 08:44:26.793097 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:26.793796 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:26.795109 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:26.795575 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:26.795696 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:26.796227 (+ 531us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 08:44:26.798399 (+ 2172us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46361/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 08:44:26.798733 (+ 334us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52174: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46361/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":47}
W20260928 08:44:26.799156 5837 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52174: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46361/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 08:44:26.801458 5863 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:44:26.791419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:32791 (local address 127.0.0.1:52174)
0928 08:44:26.792057 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:26.792081 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:26.792238 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:26.792642 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:26.792647 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:26.792659 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0928 08:44:26.792942 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:26.792950 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:26.793932 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:26.793937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:26.794975 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:26.794984 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:26.795698 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:26.795788 (+ 90us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:26.795956 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:26.795986 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 08:44:26.798633 (+ 2647us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46361/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 08:44:26.800967 (+ 2334us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:32791: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46361/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":531,"mutex_wait_us":2027,"thread_start_us":315,"threads_started":1}
I20260928 08:44:26.802476 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 5768
I20260928 08:44:26.803463 5829 generic_service.cc:196] Checking for leaks (request via RPC)
E20260928 08:44:26.811519 5864 webserver.cc:533] Webserver: sslize failed:
E20260928 08:44:26.811633 5864 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 08:44:26.893970 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 5768
2026-09-28T08:44:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (692 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T08:44:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:44:26Z Disabled control of system clock
I20260928 08:44:26.977624 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37929
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37929
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585022442112-388-0/testchainca.cert with env {}
W20260928 08:44:27.303429 5869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:27.303785 5869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:27.303963 5869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:27.311326 5869 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:44:27.311496 5869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:27.311591 5869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:27.311695 5869 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:44:27.311776 5869 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 08:44:27.311877 5869 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 08:44:27.311951 5869 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 08:44:27.321627 5869 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37929
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585022442112-388-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:27.322978 5869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:27.324285 5869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:27.335971 5875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:27.335937 5877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:27.336037 5874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:27.336898 5869 server_base.cc:1034] running on GCE node
I20260928 08:44:27.337651 5869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:27.339000 5869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:27.340215 5869 hybrid_clock.cc:648] HybridClock initialized: now 1790585067340174 us; error 53 us; skew 500 ppm
I20260928 08:44:27.340557 5869 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:44:27.342641 5869 webserver.cc:460] Webserver started at http://127.0.97.62:33973/ using document root <none> and password file <none>
I20260928 08:44:27.343286 5869 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:27.343406 5869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:27.343729 5869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:27.346515 5869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "ac81f300cff34c7daf096e3b4416118f"
format_stamp: "Formatted at 2026-09-28 08:44:27 on dist-test-slave-n6xs"
server_key: "53a88770be35c90a39ea9c3c902695bf"
server_key_iv: "8439733c70bdb5ba0cb3afb02d91578f"
server_key_version: "encryptionkey@0"
I20260928 08:44:27.347276 5869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "ac81f300cff34c7daf096e3b4416118f"
format_stamp: "Formatted at 2026-09-28 08:44:27 on dist-test-slave-n6xs"
server_key: "53a88770be35c90a39ea9c3c902695bf"
server_key_iv: "8439733c70bdb5ba0cb3afb02d91578f"
server_key_version: "encryptionkey@0"
I20260928 08:44:27.352047 5869 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260928 08:44:27.355192 5883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:27.356521 5869 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:44:27.356698 5869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "ac81f300cff34c7daf096e3b4416118f"
format_stamp: "Formatted at 2026-09-28 08:44:27 on dist-test-slave-n6xs"
server_key: "53a88770be35c90a39ea9c3c902695bf"
server_key_iv: "8439733c70bdb5ba0cb3afb02d91578f"
server_key_version: "encryptionkey@0"
I20260928 08:44:27.356885 5869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:27.369740 5869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:27.370443 5869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:27.370721 5869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:27.388063 5869 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37929
I20260928 08:44:27.388082 5934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37929 every 8 connection(s)
I20260928 08:44:27.389406 5869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:44:27.393216 5935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:27.394202 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 5869
I20260928 08:44:27.394421 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:44:27.395223 388 external_mini_cluster.cc:1428] Setting key 7982ad5a941fe32013c0b616ba0cbf95
I20260928 08:44:27.405836 5935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac81f300cff34c7daf096e3b4416118f: Bootstrap starting.
I20260928 08:44:27.408128 5938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:27.397145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34114 (local address 127.0.97.62:37929)
0928 08:44:27.398076 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:27.398087 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:27.398129 (+ 42us) server_negotiation.cc:407] Connection header received
0928 08:44:27.399717 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:27.399737 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:27.400242 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:27.400620 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:27.401135 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:27.402422 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:27.403698 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:27.404386 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:27.404581 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:27.405833 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:27.405840 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:27.405853 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:44:27.406438 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:27.406509 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:44:27.406528 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:27.406650 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:27.406878 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 08:44:27.407181 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":334,"threads_started":1}
I20260928 08:44:27.408972 5935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac81f300cff34c7daf096e3b4416118f: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:27.409986 5935 log.cc:824] T 00000000000000000000000000000000 P ac81f300cff34c7daf096e3b4416118f: Log is configured to *not* fsync() on all Append() calls
I20260928 08:44:27.411710 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:44:27.413196 5935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac81f300cff34c7daf096e3b4416118f: No bootstrap required, opened a new log
I20260928 08:44:27.422662 5935 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac81f300cff34c7daf096e3b4416118f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac81f300cff34c7daf096e3b4416118f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37929 } }
I20260928 08:44:27.423048 5935 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac81f300cff34c7daf096e3b4416118f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:27.423146 5935 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac81f300cff34c7daf096e3b4416118f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac81f300cff34c7daf096e3b4416118f, State: Initialized, Role: FOLLOWER
I20260928 08:44:27.423730 5935 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac81f300cff34c7daf096e3b4416118f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac81f300cff34c7daf096e3b4416118f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37929 } }
I20260928 08:44:27.424059 5935 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac81f300cff34c7daf096e3b4416118f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:44:27.424202 5935 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac81f300cff34c7daf096e3b4416118f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:44:27.424365 5935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac81f300cff34c7daf096e3b4416118f [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:27.426730 5935 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac81f300cff34c7daf096e3b4416118f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac81f300cff34c7daf096e3b4416118f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37929 } }
I20260928 08:44:27.427189 5935 leader_election.cc:302] T 00000000000000000000000000000000 P ac81f300cff34c7daf096e3b4416118f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac81f300cff34c7daf096e3b4416118f; no voters:
I20260928 08:44:27.427721 5935 leader_election.cc:288] T 00000000000000000000000000000000 P ac81f300cff34c7daf096e3b4416118f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:44:27.427891 5940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac81f300cff34c7daf096e3b4416118f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:44:27.428520 5940 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac81f300cff34c7daf096e3b4416118f [term 1 LEADER]: Becoming Leader. State: Replica: ac81f300cff34c7daf096e3b4416118f, State: Running, Role: LEADER
I20260928 08:44:27.429018 5940 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac81f300cff34c7daf096e3b4416118f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac81f300cff34c7daf096e3b4416118f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37929 } }
I20260928 08:44:27.429641 5935 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac81f300cff34c7daf096e3b4416118f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:44:27.432202 5941 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac81f300cff34c7daf096e3b4416118f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac81f300cff34c7daf096e3b4416118f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac81f300cff34c7daf096e3b4416118f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37929 } } }
I20260928 08:44:27.432157 5942 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac81f300cff34c7daf096e3b4416118f [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac81f300cff34c7daf096e3b4416118f. Latest consensus state: current_term: 1 leader_uuid: "ac81f300cff34c7daf096e3b4416118f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac81f300cff34c7daf096e3b4416118f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37929 } } }
I20260928 08:44:27.432619 5942 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac81f300cff34c7daf096e3b4416118f [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:27.432638 5941 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac81f300cff34c7daf096e3b4416118f [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:27.437245 5947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:44:27.442222 5947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:44:27.450850 5947 catalog_manager.cc:1347] Generated new cluster ID: af9fbbff237c42e88e5adb008285c29a
I20260928 08:44:27.450971 5947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:44:27.471498 5947 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:44:27.473354 5947 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:44:27.495471 5947 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac81f300cff34c7daf096e3b4416118f: Generated new TSK 0
I20260928 08:44:27.496567 5947 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:44:27.597337 5938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:27.513225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34126 (local address 127.0.97.62:37929)
0928 08:44:27.513396 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:27.513408 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:27.513853 (+ 445us) server_negotiation.cc:407] Connection header received
0928 08:44:27.514042 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:27.514048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:27.514146 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:27.514301 (+ 155us) server_negotiation.cc:226] Negotiated authn=JWT
0928 08:44:27.514778 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:27.515934 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:27.517263 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:27.517925 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:27.518052 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:27.518531 (+ 479us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 08:44:27.596595 (+ 78064us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 08:44:27.597017 (+ 422us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34126: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 08:44:27.597476 5938 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34126: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 08:44:27.597488 5964 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:44:27.513049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37929 (local address 127.0.0.1:34126)
0928 08:44:27.513762 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:27.513778 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:27.513889 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:27.514371 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:27.514375 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:27.514392 (+ 17us) client_negotiation.cc:189] Negotiated authn=JWT
0928 08:44:27.514636 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:27.514643 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:27.516109 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:27.516113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:27.517136 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:27.517146 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:27.518097 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:27.518137 (+ 40us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:27.518261 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:27.518292 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 08:44:27.596893 (+ 78601us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 08:44:27.597144 (+ 251us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.97.62:37929: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":427,"threads_started":1}
I20260928 08:44:27.599228 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 5869
I20260928 08:44:27.600270 5930 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:27.700913 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 5869
2026-09-28T08:44:27Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (786 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (410 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (397 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:44:28 dist-test-slave-n6xs krb5kdc[5979](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:44:28 dist-test-slave-n6xs krb5kdc[5979](info): set up 2 sockets
Sep 28 08:44:28 dist-test-slave-n6xs krb5kdc[5979](info): commencing operation
krb5kdc: starting...
W20260928 08:44:29.592433 388 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.004s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:44:29 dist-test-slave-n6xs krb5kdc[5979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:44:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:44:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:44:29.796259 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41675
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41675
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:30.114732 5992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:30.115087 5992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:30.115239 5992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:30.122313 5992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:44:30.122466 5992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:30.122573 5992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:30.122676 5992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:44:30.122761 5992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:44:30.132342 5992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:30.133716 5992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:30.135035 5992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:30.146710 6000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:30.146745 5998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:30.146710 5997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:30.147768 5992 server_base.cc:1034] running on GCE node
I20260928 08:44:30.148564 5992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:30.149962 5992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:30.151206 5992 hybrid_clock.cc:648] HybridClock initialized: now 1790585070151161 us; error 36 us; skew 500 ppm
I20260928 08:44:30.151562 5992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:30 dist-test-slave-n6xs krb5kdc[5979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:30.154675 5992 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:44:30.156183 5992 webserver.cc:460] Webserver started at http://127.0.97.62:35385/ using document root <none> and password file <none>
I20260928 08:44:30.156823 5992 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:30.156924 5992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:30.157167 5992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:30.160086 5992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "805a166b23834aaa8b5dd8b1899642e2"
format_stamp: "Formatted at 2026-09-28 08:44:30 on dist-test-slave-n6xs"
server_key: "e3211d9349d0984d6c556f2cd3893267"
server_key_iv: "5eeae81869fe8cfbf28b2bffaa5bc83d"
server_key_version: "encryptionkey@0"
I20260928 08:44:30.160796 5992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "805a166b23834aaa8b5dd8b1899642e2"
format_stamp: "Formatted at 2026-09-28 08:44:30 on dist-test-slave-n6xs"
server_key: "e3211d9349d0984d6c556f2cd3893267"
server_key_iv: "5eeae81869fe8cfbf28b2bffaa5bc83d"
server_key_version: "encryptionkey@0"
I20260928 08:44:30.165995 5992 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 08:44:30.169431 6007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:30.170843 5992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:44:30.171085 5992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "805a166b23834aaa8b5dd8b1899642e2"
format_stamp: "Formatted at 2026-09-28 08:44:30 on dist-test-slave-n6xs"
server_key: "e3211d9349d0984d6c556f2cd3893267"
server_key_iv: "5eeae81869fe8cfbf28b2bffaa5bc83d"
server_key_version: "encryptionkey@0"
I20260928 08:44:30.171271 5992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:30.191368 5992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:30.195732 5992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:30.196048 5992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:30.214432 5992 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41675
I20260928 08:44:30.214452 6058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41675 every 8 connection(s)
I20260928 08:44:30.215878 5992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:44:30.219473 6059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:30.224534 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 5992
I20260928 08:44:30.224751 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:44:30.225361 388 external_mini_cluster.cc:1428] Setting key c90b37b963fab267467f4506f9a3184d
I20260928 08:44:30.230022 6059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 805a166b23834aaa8b5dd8b1899642e2: Bootstrap starting.
I20260928 08:44:30.233676 6059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 805a166b23834aaa8b5dd8b1899642e2: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:30.234942 6059 log.cc:824] T 00000000000000000000000000000000 P 805a166b23834aaa8b5dd8b1899642e2: Log is configured to *not* fsync() on all Append() calls
I20260928 08:44:30.238183 6059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 805a166b23834aaa8b5dd8b1899642e2: No bootstrap required, opened a new log
Sep 28 08:44:30 dist-test-slave-n6xs krb5kdc[5979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:44:30.246181 6059 raft_consensus.cc:348] T 00000000000000000000000000000000 P 805a166b23834aaa8b5dd8b1899642e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "805a166b23834aaa8b5dd8b1899642e2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41675 } }
I20260928 08:44:30.246582 6059 raft_consensus.cc:374] T 00000000000000000000000000000000 P 805a166b23834aaa8b5dd8b1899642e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:30.246701 6059 raft_consensus.cc:729] T 00000000000000000000000000000000 P 805a166b23834aaa8b5dd8b1899642e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 805a166b23834aaa8b5dd8b1899642e2, State: Initialized, Role: FOLLOWER
I20260928 08:44:30.247393 6059 consensus_queue.cc:260] T 00000000000000000000000000000000 P 805a166b23834aaa8b5dd8b1899642e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "805a166b23834aaa8b5dd8b1899642e2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41675 } }
I20260928 08:44:30.247643 6059 raft_consensus.cc:388] T 00000000000000000000000000000000 P 805a166b23834aaa8b5dd8b1899642e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:44:30.247772 6059 raft_consensus.cc:482] T 00000000000000000000000000000000 P 805a166b23834aaa8b5dd8b1899642e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:44:30.247997 6059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 805a166b23834aaa8b5dd8b1899642e2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:30.248793 6062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:30.227403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39720 (local address 127.0.97.62:41675)
0928 08:44:30.228297 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:30.228309 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:30.228349 (+ 40us) server_negotiation.cc:407] Connection header received
0928 08:44:30.229993 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:30.230020 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:30.230441 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:30.230764 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:30.232547 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:30.233884 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:30.235629 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:30.236605 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:30.236732 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:30.241225 (+ 4493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:30.241247 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:30.241262 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:30.241300 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:30.244395 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:30.245099 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:30.245109 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:30.245115 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:30.245272 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:30.245621 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:30.245627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:30.245631 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:30.246141 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:30.246422 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:30.246871 (+ 449us) server_negotiation.cc:299] Negotiation successful
0928 08:44:30.247292 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":424,"threads_started":1}
I20260928 08:44:30.250603 6059 raft_consensus.cc:504] T 00000000000000000000000000000000 P 805a166b23834aaa8b5dd8b1899642e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "805a166b23834aaa8b5dd8b1899642e2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41675 } }
I20260928 08:44:30.251106 6059 leader_election.cc:302] T 00000000000000000000000000000000 P 805a166b23834aaa8b5dd8b1899642e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 805a166b23834aaa8b5dd8b1899642e2; no voters:
I20260928 08:44:30.251763 6059 leader_election.cc:288] T 00000000000000000000000000000000 P 805a166b23834aaa8b5dd8b1899642e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:44:30.251909 6064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 805a166b23834aaa8b5dd8b1899642e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:44:30.252565 6064 raft_consensus.cc:686] T 00000000000000000000000000000000 P 805a166b23834aaa8b5dd8b1899642e2 [term 1 LEADER]: Becoming Leader. State: Replica: 805a166b23834aaa8b5dd8b1899642e2, State: Running, Role: LEADER
I20260928 08:44:30.253134 6064 consensus_queue.cc:237] T 00000000000000000000000000000000 P 805a166b23834aaa8b5dd8b1899642e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "805a166b23834aaa8b5dd8b1899642e2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41675 } }
I20260928 08:44:30.254025 6059 sys_catalog.cc:564] T 00000000000000000000000000000000 P 805a166b23834aaa8b5dd8b1899642e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:44:30.255781 6065 sys_catalog.cc:455] T 00000000000000000000000000000000 P 805a166b23834aaa8b5dd8b1899642e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "805a166b23834aaa8b5dd8b1899642e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "805a166b23834aaa8b5dd8b1899642e2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41675 } } }
I20260928 08:44:30.256184 6065 sys_catalog.cc:458] T 00000000000000000000000000000000 P 805a166b23834aaa8b5dd8b1899642e2 [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:30.256552 6066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 805a166b23834aaa8b5dd8b1899642e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 805a166b23834aaa8b5dd8b1899642e2. Latest consensus state: current_term: 1 leader_uuid: "805a166b23834aaa8b5dd8b1899642e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "805a166b23834aaa8b5dd8b1899642e2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41675 } } }
I20260928 08:44:30.256847 6066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 805a166b23834aaa8b5dd8b1899642e2 [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:30.259021 6070 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:44:30.263602 6070 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:44:30.271937 6070 catalog_manager.cc:1347] Generated new cluster ID: ec9a99cded47454590f513bec8766278
I20260928 08:44:30.272081 6070 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:44:30.286365 6070 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:44:30.288182 6070 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:44:30.300272 6070 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 805a166b23834aaa8b5dd8b1899642e2: Generated new TSK 0
I20260928 08:44:30.301326 6070 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:44:30.355623 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41675
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:30.678700 6085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:30.678970 6085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:30.679064 6085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:30.687178 6085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:30.687299 6085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:30.687417 6085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:44:30.697311 6085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:30.698529 6085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:30.699965 6085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:30.712952 6090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:30.712980 6091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:30.713126 6093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:30.713747 6085 server_base.cc:1034] running on GCE node
I20260928 08:44:30.714286 6085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:30.715189 6085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:30.716403 6085 hybrid_clock.cc:648] HybridClock initialized: now 1790585070716353 us; error 44 us; skew 500 ppm
I20260928 08:44:30.716717 6085 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:30 dist-test-slave-n6xs krb5kdc[5979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:30.719782 6085 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:44:30.721189 6085 webserver.cc:460] Webserver started at http://127.0.97.1:42995/ using document root <none> and password file <none>
I20260928 08:44:30.721868 6085 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:30.722005 6085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:30.722307 6085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:30.725174 6085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-0/data/instance:
uuid: "ee23abd9f77a4466af2273ce052141d3"
format_stamp: "Formatted at 2026-09-28 08:44:30 on dist-test-slave-n6xs"
server_key: "e0671ca9f0ce96477fa5888b450204b8"
server_key_iv: "4573a69f3fc81efd5294ce1623431cdc"
server_key_version: "encryptionkey@0"
I20260928 08:44:30.725929 6085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-0/wal/instance:
uuid: "ee23abd9f77a4466af2273ce052141d3"
format_stamp: "Formatted at 2026-09-28 08:44:30 on dist-test-slave-n6xs"
server_key: "e0671ca9f0ce96477fa5888b450204b8"
server_key_iv: "4573a69f3fc81efd5294ce1623431cdc"
server_key_version: "encryptionkey@0"
I20260928 08:44:30.730845 6085 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:44:30.734143 6100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:30.735451 6085 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260928 08:44:30.735620 6085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-0/wal
uuid: "ee23abd9f77a4466af2273ce052141d3"
format_stamp: "Formatted at 2026-09-28 08:44:30 on dist-test-slave-n6xs"
server_key: "e0671ca9f0ce96477fa5888b450204b8"
server_key_iv: "4573a69f3fc81efd5294ce1623431cdc"
server_key_version: "encryptionkey@0"
I20260928 08:44:30.735874 6085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:30.756239 6085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:30.760584 6085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:30.760856 6085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:30.761766 6085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:44:30.763032 6085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:44:30.763134 6085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:30.763268 6085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:44:30.763335 6085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:30.796595 6085 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:41743
I20260928 08:44:30.796643 6212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:41743 every 8 connection(s)
I20260928 08:44:30.797820 6085 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:44:30.805917 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 6085
I20260928 08:44:30.806118 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:44:30.806710 388 external_mini_cluster.cc:1428] Setting key ca4d3683dae4bc6d558fa2a16f282e92
Sep 28 08:44:30 dist-test-slave-n6xs krb5kdc[5979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:44:30.817796 6215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:30.800523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:32793 (local address 127.0.97.62:41675)
0928 08:44:30.800936 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:30.800940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:30.801857 (+ 917us) server_negotiation.cc:407] Connection header received
0928 08:44:30.803742 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:30.803747 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:30.803846 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:30.803967 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:30.805366 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:30.806146 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:30.807769 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:30.808255 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:30.808369 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:30.811638 (+ 3269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:30.811647 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:30.811655 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:30.811698 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:30.814695 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:30.815614 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:30.815618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:30.815620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:30.815736 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:30.816210 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:30.816213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:30.816215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:30.816462 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:30.816577 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:30.817390 (+ 813us) server_negotiation.cc:299] Negotiation successful
0928 08:44:30.817554 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":222,"threads_started":1}
I20260928 08:44:30.819301 6216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:30.801029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41675 (local address 127.0.97.1:32793)
0928 08:44:30.801705 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:30.801742 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:30.803486 (+ 1744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:30.804197 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:30.804204 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:30.804670 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:30.805202 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:30.805218 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:30.806299 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:30.806303 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:30.807635 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:30.807649 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:30.808474 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:30.808484 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:30.808699 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:30.809360 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:30.809382 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:30.811424 (+ 2042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:30.814842 (+ 3418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:30.814852 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:30.814868 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:30.815508 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:30.815866 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:30.815874 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:30.815881 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:30.816109 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:30.816589 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:30.816599 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:30.817027 (+ 428us) client_negotiation.cc:769] Sending connection context
0928 08:44:30.817333 (+ 306us) client_negotiation.cc:240] Negotiation successful
0928 08:44:30.817894 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":268,"threads_started":1}
I20260928 08:44:30.820875 6213 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41675
I20260928 08:44:30.821197 6213 heartbeater.cc:464] Registering TS with master...
I20260928 08:44:30.821869 6213 heartbeater.cc:511] Master 127.0.97.62:41675 requested a full tablet report, sending...
I20260928 08:44:30.824191 6024 ts_manager.cc:194] Registered new tserver with Master: ee23abd9f77a4466af2273ce052141d3 (127.0.97.1:41743)
I20260928 08:44:30.826387 6024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:32793
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 08:44:30.855077 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41675
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:31.169719 6219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:31.170013 6219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:31.170111 6219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:31.177204 6219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:31.177315 6219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:31.177428 6219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:44:31.186911 6219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:31.188148 6219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:31.189356 6219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:31.202342 6224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:31.202267 6227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:31.202306 6225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:31.203604 6219 server_base.cc:1034] running on GCE node
I20260928 08:44:31.204229 6219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:31.205174 6219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:31.206368 6219 hybrid_clock.cc:648] HybridClock initialized: now 1790585071206302 us; error 63 us; skew 500 ppm
I20260928 08:44:31.206687 6219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:31 dist-test-slave-n6xs krb5kdc[5979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:31.215327 6219 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:44:31.216945 6219 webserver.cc:460] Webserver started at http://127.0.97.2:40777/ using document root <none> and password file <none>
I20260928 08:44:31.217614 6219 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:31.217736 6219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:31.218024 6219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:31.220880 6219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-1/data/instance:
uuid: "53727bcd6a444c728ae3ed53dd7f4b47"
format_stamp: "Formatted at 2026-09-28 08:44:31 on dist-test-slave-n6xs"
server_key: "2c45a5baa0653f1c55a2525afdddca99"
server_key_iv: "a87abe8cd7f1248414293276bd1db582"
server_key_version: "encryptionkey@0"
I20260928 08:44:31.221627 6219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-1/wal/instance:
uuid: "53727bcd6a444c728ae3ed53dd7f4b47"
format_stamp: "Formatted at 2026-09-28 08:44:31 on dist-test-slave-n6xs"
server_key: "2c45a5baa0653f1c55a2525afdddca99"
server_key_iv: "a87abe8cd7f1248414293276bd1db582"
server_key_version: "encryptionkey@0"
I20260928 08:44:31.226637 6219 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:44:31.230050 6234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:31.231391 6219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:44:31.231580 6219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-1/wal
uuid: "53727bcd6a444c728ae3ed53dd7f4b47"
format_stamp: "Formatted at 2026-09-28 08:44:31 on dist-test-slave-n6xs"
server_key: "2c45a5baa0653f1c55a2525afdddca99"
server_key_iv: "a87abe8cd7f1248414293276bd1db582"
server_key_version: "encryptionkey@0"
I20260928 08:44:31.231751 6219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:31.245316 6219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:31.249408 6219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:31.249688 6219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:31.250595 6219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:44:31.251914 6219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:44:31.252007 6219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:31.252146 6219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:44:31.252210 6219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:31.284932 6219 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:32913
I20260928 08:44:31.285014 6346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:32913 every 8 connection(s)
I20260928 08:44:31.286192 6219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:44:31.293599 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 6219
I20260928 08:44:31.293805 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:44:31.294207 388 external_mini_cluster.cc:1428] Setting key 066f8f908a4f15367f887870d7f7e0b3
Sep 28 08:44:31 dist-test-slave-n6xs krb5kdc[5979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:44:31.308043 6215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:31.288806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49043 (local address 127.0.97.62:41675)
0928 08:44:31.288963 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:31.288967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:31.289960 (+ 993us) server_negotiation.cc:407] Connection header received
0928 08:44:31.291742 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:31.291748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:31.291860 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:31.291982 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:31.293506 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:31.294365 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:31.296239 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:31.296733 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:31.296835 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:31.301531 (+ 4696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:31.301538 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:31.301542 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:31.301578 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:31.304346 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:31.305398 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:31.305401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:31.305404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:31.305524 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:31.306100 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:31.306103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:31.306105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:31.306348 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:31.306453 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:31.307428 (+ 975us) server_negotiation.cc:299] Negotiation successful
0928 08:44:31.307660 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 08:44:31.309787 6349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:31.289145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41675 (local address 127.0.97.2:49043)
0928 08:44:31.289815 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:31.289852 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:31.291487 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:31.292217 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:31.292224 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:31.292693 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:31.293303 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:31.293328 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:31.294520 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:31.294526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:31.296018 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:31.296030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:31.296957 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:31.296964 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:31.297334 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:31.298112 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:31.298153 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:31.301263 (+ 3110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:31.304499 (+ 3236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:31.304512 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:31.304535 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:31.305272 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:31.305639 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:31.305649 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:31.305657 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:31.305982 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:31.306480 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:31.306493 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:31.307011 (+ 518us) client_negotiation.cc:769] Sending connection context
0928 08:44:31.307385 (+ 374us) client_negotiation.cc:240] Negotiation successful
0928 08:44:31.308085 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":265,"threads_started":1}
I20260928 08:44:31.311426 6347 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41675
I20260928 08:44:31.311745 6347 heartbeater.cc:464] Registering TS with master...
I20260928 08:44:31.312592 6347 heartbeater.cc:511] Master 127.0.97.62:41675 requested a full tablet report, sending...
I20260928 08:44:31.314246 6024 ts_manager.cc:194] Registered new tserver with Master: 53727bcd6a444c728ae3ed53dd7f4b47 (127.0.97.2:32913)
I20260928 08:44:31.315074 6024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:49043
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:44:31.344825 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41675
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:40663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:31.661054 6352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:31.661309 6352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:31.661409 6352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:31.667868 6352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:31.667955 6352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:31.668057 6352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:44:31.677129 6352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:40663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:31.678264 6352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:31.679476 6352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:31.692560 6358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:31.692780 6357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:31.692575 6360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:31.693873 6352 server_base.cc:1034] running on GCE node
I20260928 08:44:31.694505 6352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:31.695382 6352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:31.696606 6352 hybrid_clock.cc:648] HybridClock initialized: now 1790585071696563 us; error 57 us; skew 500 ppm
I20260928 08:44:31.696950 6352 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:31 dist-test-slave-n6xs krb5kdc[5979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:31.700562 6352 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:44:31.702147 6352 webserver.cc:460] Webserver started at http://127.0.97.3:42183/ using document root <none> and password file <none>
I20260928 08:44:31.702873 6352 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:31.703027 6352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:31.703332 6352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:31.706295 6352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-2/data/instance:
uuid: "9e43fab91efe4ddc87b05c2e8deca413"
format_stamp: "Formatted at 2026-09-28 08:44:31 on dist-test-slave-n6xs"
server_key: "4a9c0502ff0e71d4499522229b656725"
server_key_iv: "c45cc7d1383f19cd7a3091e51c70dc6f"
server_key_version: "encryptionkey@0"
I20260928 08:44:31.707100 6352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-2/wal/instance:
uuid: "9e43fab91efe4ddc87b05c2e8deca413"
format_stamp: "Formatted at 2026-09-28 08:44:31 on dist-test-slave-n6xs"
server_key: "4a9c0502ff0e71d4499522229b656725"
server_key_iv: "c45cc7d1383f19cd7a3091e51c70dc6f"
server_key_version: "encryptionkey@0"
I20260928 08:44:31.712234 6352 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 08:44:31.715580 6367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:31.716852 6352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:44:31.717033 6352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-2/wal
uuid: "9e43fab91efe4ddc87b05c2e8deca413"
format_stamp: "Formatted at 2026-09-28 08:44:31 on dist-test-slave-n6xs"
server_key: "4a9c0502ff0e71d4499522229b656725"
server_key_iv: "c45cc7d1383f19cd7a3091e51c70dc6f"
server_key_version: "encryptionkey@0"
I20260928 08:44:31.717234 6352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:31.734659 6352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:31.739353 6352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:31.739711 6352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:31.740726 6352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:44:31.742122 6352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:44:31.742231 6352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:31.742383 6352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:44:31.742468 6352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:31.777441 6352 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39001
I20260928 08:44:31.777458 6479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39001 every 8 connection(s)
I20260928 08:44:31.778813 6352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:44:31.783911 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 6352
I20260928 08:44:31.784086 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:44:31.784487 388 external_mini_cluster.cc:1428] Setting key 60b62f28d5245bfe63bf0808b14f4d0f
Sep 28 08:44:31 dist-test-slave-n6xs krb5kdc[5979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:44:31.799041 6215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:31.781279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48799 (local address 127.0.97.62:41675)
0928 08:44:31.781473 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:31.781479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:31.782690 (+ 1211us) server_negotiation.cc:407] Connection header received
0928 08:44:31.784705 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:31.784711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:31.784797 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:31.784913 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:31.786699 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:31.787693 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:31.789240 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:31.789625 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:31.789746 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:31.793060 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:31.793065 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:31.793069 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:31.793098 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:31.795599 (+ 2501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:31.796418 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:31.796422 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:31.796425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:31.796547 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:31.796998 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:31.797001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:31.797002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:31.797351 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:31.797497 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:31.798441 (+ 944us) server_negotiation.cc:299] Negotiation successful
0928 08:44:31.798643 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 08:44:31.799958 6482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:31.781782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41675 (local address 127.0.97.3:48799)
0928 08:44:31.782530 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:31.782579 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:31.784354 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:31.785285 (+ 931us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:31.785302 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:31.785895 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:31.786545 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:31.786564 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:31.787919 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:31.787922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:31.789107 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:31.789118 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:31.789831 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:31.789838 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:31.790057 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:31.790700 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:31.790721 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:31.792879 (+ 2158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:31.795710 (+ 2831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:31.795715 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:31.795725 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:31.796272 (+ 547us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:31.796651 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:31.796656 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:31.796659 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:31.796879 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:31.797487 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:31.797494 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:31.797873 (+ 379us) client_negotiation.cc:769] Sending connection context
0928 08:44:31.798195 (+ 322us) client_negotiation.cc:240] Negotiation successful
0928 08:44:31.798703 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":291,"threads_started":1}
I20260928 08:44:31.801196 6480 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41675
I20260928 08:44:31.801492 6480 heartbeater.cc:464] Registering TS with master...
I20260928 08:44:31.802155 6480 heartbeater.cc:511] Master 127.0.97.62:41675 requested a full tablet report, sending...
I20260928 08:44:31.803493 6024 ts_manager.cc:194] Registered new tserver with Master: 9e43fab91efe4ddc87b05c2e8deca413 (127.0.97.3:39001)
I20260928 08:44:31.804328 6024 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:48799
I20260928 08:44:31.811468 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:44:31.829521 6213 heartbeater.cc:503] Master 127.0.97.62:41675 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/test-admin.keytab.
Sep 28 08:44:31 dist-test-slave-n6xs krb5kdc[5979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:31.836951 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 08:44:31.842082 6487 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:44:31.839705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51938 (local address 127.0.97.1:41743)
0928 08:44:31.840391 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:31.840398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:31.840433 (+ 35us) server_negotiation.cc:407] Connection header received
0928 08:44:31.840488 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:31.840494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:31.840709 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:31.840854 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:31.841647 (+ 793us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51938: BlockingRecv error: recv got EOF from 127.0.0.1:51938 (error 108)
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":395,"threads_started":1}
W20260928 08:44:31.842392 388 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/krb5cc)
Sep 28 08:44:31 dist-test-slave-n6xs krb5kdc[5979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:31.843997 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 08:44:31 dist-test-slave-n6xs krb5kdc[5979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:31.845429 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 08:44:31 dist-test-slave-n6xs krb5kdc[5979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:44:31.860466 6487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:31.847031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51952 (local address 127.0.97.1:41743)
0928 08:44:31.847216 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:31.847221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:31.847278 (+ 57us) server_negotiation.cc:407] Connection header received
0928 08:44:31.847409 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:31.847413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:31.847463 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:31.847594 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:31.848606 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:31.849506 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:31.850689 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:31.851230 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:31.851366 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:31.854638 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:31.854648 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:31.854668 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:31.854700 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:31.857490 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:31.858124 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:31.858133 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:31.858135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:31.858259 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:31.858679 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:31.858690 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:31.858693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:31.859186 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:31.859502 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:31.859872 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 08:44:31.860129 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 08:44:31.865990 6487 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:44:31.864351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51968 (local address 127.0.97.1:41743)
0928 08:44:31.864536 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:31.864543 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:31.864815 (+ 272us) server_negotiation.cc:407] Connection header received
0928 08:44:31.864976 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:31.864985 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:31.865035 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:31.865145 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:31.865758 (+ 613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51968: BlockingRecv error: recv got EOF from 127.0.0.1:51968 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 08:44:31.866238 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/krb5cc' of type FILE
Sep 28 08:44:31 dist-test-slave-n6xs krb5kdc[5979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:31.867784 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 08:44:31 dist-test-slave-n6xs krb5kdc[5979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:31.869019 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 08:44:31 dist-test-slave-n6xs krb5kdc[5979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:44:31.882321 6487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:31.870371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51976 (local address 127.0.97.1:41743)
0928 08:44:31.870552 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:31.870557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:31.870712 (+ 155us) server_negotiation.cc:407] Connection header received
0928 08:44:31.870873 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:31.870878 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:31.870939 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:31.871029 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:31.871938 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:31.872638 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:31.873806 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:31.874231 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:31.874353 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:31.877715 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:31.877720 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:31.877723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:31.877750 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:31.880062 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:31.880643 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:31.880650 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:31.880653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:31.880768 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:31.881113 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:31.881118 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:31.881121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:31.881368 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:31.881611 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:31.881863 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 08:44:31.882042 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 08:44:31.886435 6487 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:44:31.884812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51990 (local address 127.0.97.1:41743)
0928 08:44:31.884990 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:31.884995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:31.885222 (+ 227us) server_negotiation.cc:407] Connection header received
0928 08:44:31.885339 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:31.885342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:31.885403 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:31.885483 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:31.886204 (+ 721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51990: BlockingRecv error: recv got EOF from 127.0.0.1:51990 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 08:44:31.886535 388 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790585022442112-388-0/krb5kdc/krb5cc' of type FILE
Sep 28 08:44:31 dist-test-slave-n6xs krb5kdc[5979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:31.888031 388 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 08:44:31 dist-test-slave-n6xs krb5kdc[5979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:31.889235 388 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 08:44:31 dist-test-slave-n6xs krb5kdc[5979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:44:31.902659 6487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:31.890418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52004 (local address 127.0.97.1:41743)
0928 08:44:31.890622 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:31.890629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:31.890833 (+ 204us) server_negotiation.cc:407] Connection header received
0928 08:44:31.891043 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:31.891048 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:31.891107 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:31.891207 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:31.892158 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:31.892847 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:31.894042 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:31.894445 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:31.894566 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:31.897800 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:31.897804 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:31.897808 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:31.897834 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:31.900391 (+ 2557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:31.900963 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:31.900967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:31.900969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:31.901099 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:31.901519 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:31.901523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:31.901525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:31.901772 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:31.902017 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:31.902228 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 08:44:31.902416 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 08:44:31.904067 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 6085
I20260928 08:44:31.914705 6487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:31.904666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52006 (local address 127.0.97.1:41743)
0928 08:44:31.904854 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:31.904861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:31.905112 (+ 251us) server_negotiation.cc:407] Connection header received
0928 08:44:31.905212 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:31.905215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:31.905262 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:31.905333 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:31.906095 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:31.906730 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:31.907917 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:31.908318 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:31.908402 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:31.910180 (+ 1778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:31.910184 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:31.910187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:31.910212 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:31.912430 (+ 2218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:31.913067 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:31.913070 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:31.913072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:31.913176 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:31.913477 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:31.913479 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:31.913481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:31.913707 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:31.913872 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:31.914229 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 08:44:31.914369 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 08:44:31.915334 6208 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:32.000849 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 6085
I20260928 08:44:32.016893 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 6219
Sep 28 08:44:32 dist-test-slave-n6xs krb5kdc[5979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 08:44:32.033515 6500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:32.017820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48618 (local address 127.0.97.2:32913)
0928 08:44:32.018444 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:32.018458 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:32.018484 (+ 26us) server_negotiation.cc:407] Connection header received
0928 08:44:32.018545 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:32.018552 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:32.018790 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:32.018990 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:32.020048 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:32.021285 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:32.022650 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:32.023250 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:32.023409 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:32.027338 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:32.027348 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:32.027362 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:32.027395 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:32.030503 (+ 3108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:32.031217 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:32.031232 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:32.031235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:32.031360 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:32.031775 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:32.031782 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:32.031785 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:32.032256 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:32.032596 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:32.032913 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 08:44:32.033179 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":342,"threads_started":1}
I20260928 08:44:32.034842 6342 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:32.116726 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 6219
I20260928 08:44:32.132237 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 6352
Sep 28 08:44:32 dist-test-slave-n6xs krb5kdc[5979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 08:44:32.147573 6502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:32.133210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56366 (local address 127.0.97.3:39001)
0928 08:44:32.133846 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:32.133853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:32.133880 (+ 27us) server_negotiation.cc:407] Connection header received
0928 08:44:32.133998 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:32.134005 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:32.134213 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:32.134385 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:32.135425 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:32.136476 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:32.137704 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:32.138246 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:32.138376 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:32.142240 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:32.142250 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:32.142264 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:32.142292 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:32.145136 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:32.145741 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:32.145749 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:32.145757 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:32.145863 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:32.146204 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:32.146210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:32.146213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:32.146523 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:32.146832 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:32.147062 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 08:44:32.147286 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":355,"threads_started":1}
I20260928 08:44:32.148933 6475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:32.231400 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 6352
I20260928 08:44:32.247104 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 5992
I20260928 08:44:32.248317 6054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:32.350052 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 5992
2026-09-28T08:44:32Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3846 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:44:32 dist-test-slave-n6xs krb5kdc[6507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:44:32 dist-test-slave-n6xs krb5kdc[6507](info): set up 2 sockets
Sep 28 08:44:32 dist-test-slave-n6xs krb5kdc[6507](info): commencing operation
krb5kdc: starting...
W20260928 08:44:33.481114 388 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:44:33 dist-test-slave-n6xs krb5kdc[6507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:44:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:44:33Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260928 08:44:33.691458 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46083
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:34.007612 6520 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:34.008041 6520 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:34.008239 6520 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:34.015683 6520 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:44:34.015869 6520 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:34.015986 6520 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:34.016096 6520 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:44:34.016191 6520 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:44:34.026417 6520 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:34.027798 6520 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:34.029178 6520 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:34.040695 6525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:34.040774 6526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:34.040840 6528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:34.041860 6520 server_base.cc:1034] running on GCE node
I20260928 08:44:34.042605 6520 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:34.044385 6520 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:34.045624 6520 hybrid_clock.cc:648] HybridClock initialized: now 1790585074045584 us; error 59 us; skew 500 ppm
I20260928 08:44:34.046002 6520 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:34 dist-test-slave-n6xs krb5kdc[6507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585074, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:34.049266 6520 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260928 08:44:34.050637 6520 webserver.cc:460] Webserver started at http://127.0.97.62:41359/ using document root <none> and password file <none>
I20260928 08:44:34.051342 6520 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:34.051548 6520 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:34.051944 6520 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:34.054804 6520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "0c813b2df2b645b18971acadaa4ca63a"
format_stamp: "Formatted at 2026-09-28 08:44:34 on dist-test-slave-n6xs"
server_key: "62f71f83f7276a51628bb90bba9daf1b"
server_key_iv: "067f2c5daf5bf71b7aa23891ea817231"
server_key_version: "encryptionkey@0"
I20260928 08:44:34.055593 6520 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "0c813b2df2b645b18971acadaa4ca63a"
format_stamp: "Formatted at 2026-09-28 08:44:34 on dist-test-slave-n6xs"
server_key: "62f71f83f7276a51628bb90bba9daf1b"
server_key_iv: "067f2c5daf5bf71b7aa23891ea817231"
server_key_version: "encryptionkey@0"
I20260928 08:44:34.060683 6520 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 08:44:34.064432 6535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:34.065738 6520 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 08:44:34.065867 6520 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "0c813b2df2b645b18971acadaa4ca63a"
format_stamp: "Formatted at 2026-09-28 08:44:34 on dist-test-slave-n6xs"
server_key: "62f71f83f7276a51628bb90bba9daf1b"
server_key_iv: "067f2c5daf5bf71b7aa23891ea817231"
server_key_version: "encryptionkey@0"
I20260928 08:44:34.066015 6520 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:34.076594 6520 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:34.080703 6520 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:34.080916 6520 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:34.098953 6520 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46083
I20260928 08:44:34.099046 6586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46083 every 8 connection(s)
I20260928 08:44:34.100361 6520 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:44:34.104115 6587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:34.110248 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 6520
I20260928 08:44:34.110531 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:44:34.111313 388 external_mini_cluster.cc:1428] Setting key 48dd35a9dd0d407b48a1932190b78531
I20260928 08:44:34.114283 6587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a: Bootstrap starting.
I20260928 08:44:34.117427 6587 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:34.118803 6587 log.cc:824] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a: Log is configured to *not* fsync() on all Append() calls
I20260928 08:44:34.121979 6587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a: No bootstrap required, opened a new log
Sep 28 08:44:34 dist-test-slave-n6xs krb5kdc[6507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 08:44:34.131062 6587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } }
I20260928 08:44:34.131417 6587 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:34.131516 6587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c813b2df2b645b18971acadaa4ca63a, State: Initialized, Role: FOLLOWER
I20260928 08:44:34.132185 6587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } }
I20260928 08:44:34.132448 6587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:44:34.132607 6587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:44:34.132802 6587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:34.135777 6590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:34.113663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35920 (local address 127.0.97.62:46083)
0928 08:44:34.114613 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:34.114625 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:34.114669 (+ 44us) server_negotiation.cc:407] Connection header received
0928 08:44:34.116688 (+ 2019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:34.116711 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:34.117112 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:34.117490 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:34.119392 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:34.121060 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:34.122867 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:34.123525 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:34.123652 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:34.128140 (+ 4488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:34.128147 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:34.128164 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:34.128195 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:34.131139 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:34.131942 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:34.131949 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:34.131955 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:34.132110 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:34.132605 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:34.132611 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:34.132615 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:34.133353 (+ 738us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:34.133676 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:34.134028 (+ 352us) server_negotiation.cc:299] Negotiation successful
0928 08:44:34.134474 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":459,"threads_started":1}
I20260928 08:44:34.136020 6587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } }
I20260928 08:44:34.136591 6587 leader_election.cc:302] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c813b2df2b645b18971acadaa4ca63a; no voters:
I20260928 08:44:34.137279 6587 leader_election.cc:288] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:44:34.137465 6592 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:44:34.139163 6587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:44:34.139523 6592 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [term 1 LEADER]: Becoming Leader. State: Replica: 0c813b2df2b645b18971acadaa4ca63a, State: Running, Role: LEADER
I20260928 08:44:34.140173 6592 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } }
I20260928 08:44:34.143399 6594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c813b2df2b645b18971acadaa4ca63a. Latest consensus state: current_term: 1 leader_uuid: "0c813b2df2b645b18971acadaa4ca63a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } }
I20260928 08:44:34.143887 6594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:34.143536 6593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c813b2df2b645b18971acadaa4ca63a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } }
I20260928 08:44:34.144090 6593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:34.147552 6604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:44:34.151641 6604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:44:34.159241 6604 catalog_manager.cc:1347] Generated new cluster ID: cd79b6032ebe422ab25f6fc91d390632
I20260928 08:44:34.159353 6604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:44:34.175375 6604 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:44:34.176972 6604 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:44:34.186563 6604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a: Generated new TSK 0
I20260928 08:44:34.187347 6604 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260928 08:44:34.248389 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46083
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:34.567408 6613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:34.567724 6613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:34.567914 6613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:34.575273 6613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:34.575443 6613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:34.575628 6613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:44:34.585466 6613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46083
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:34.586717 6613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:34.588001 6613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:34.600682 6618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:34.600615 6621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:34.600656 6619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:34.601379 6613 server_base.cc:1034] running on GCE node
I20260928 08:44:34.601888 6613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:34.602897 6613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:34.604126 6613 hybrid_clock.cc:648] HybridClock initialized: now 1790585074604073 us; error 47 us; skew 500 ppm
I20260928 08:44:34.604432 6613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:34 dist-test-slave-n6xs krb5kdc[6507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585074, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:34.607647 6613 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260928 08:44:34.609190 6613 webserver.cc:460] Webserver started at http://127.0.97.1:41501/ using document root <none> and password file <none>
I20260928 08:44:34.609846 6613 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:34.609979 6613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:34.610273 6613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:34.613098 6613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-0/data/instance:
uuid: "fb125058c6c34aae8dff092e2f334d3b"
format_stamp: "Formatted at 2026-09-28 08:44:34 on dist-test-slave-n6xs"
server_key: "6f17a249f864445954cc0ebb8c947195"
server_key_iv: "c039a0acfef6d4227718eefdf66ebaca"
server_key_version: "encryptionkey@0"
I20260928 08:44:34.613849 6613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-0/wal/instance:
uuid: "fb125058c6c34aae8dff092e2f334d3b"
format_stamp: "Formatted at 2026-09-28 08:44:34 on dist-test-slave-n6xs"
server_key: "6f17a249f864445954cc0ebb8c947195"
server_key_iv: "c039a0acfef6d4227718eefdf66ebaca"
server_key_version: "encryptionkey@0"
I20260928 08:44:34.618786 6613 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:44:34.622056 6628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:34.623306 6613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 08:44:34.623497 6613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-0/wal
uuid: "fb125058c6c34aae8dff092e2f334d3b"
format_stamp: "Formatted at 2026-09-28 08:44:34 on dist-test-slave-n6xs"
server_key: "6f17a249f864445954cc0ebb8c947195"
server_key_iv: "c039a0acfef6d4227718eefdf66ebaca"
server_key_version: "encryptionkey@0"
I20260928 08:44:34.623683 6613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:34.658808 6613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:34.663118 6613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:34.663398 6613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:34.664371 6613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:44:34.665627 6613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:44:34.665723 6613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:34.665872 6613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:44:34.665951 6613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:34.698681 6613 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46847
I20260928 08:44:34.698726 6740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46847 every 8 connection(s)
I20260928 08:44:34.700037 6613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:44:34.700349 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 6613
I20260928 08:44:34.700562 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:44:34.701433 388 external_mini_cluster.cc:1428] Setting key 453d8863d24e6e737ee62491a6be5bbf
Sep 28 08:44:34 dist-test-slave-n6xs krb5kdc[6507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585074, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 08:44:34.723050 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:34.702697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39577 (local address 127.0.97.62:46083)
0928 08:44:34.703254 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:34.703261 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:34.704636 (+ 1375us) server_negotiation.cc:407] Connection header received
0928 08:44:34.707477 (+ 2841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:34.707481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:34.707547 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:34.707713 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:34.709328 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:34.710087 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:34.711700 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:34.712190 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:34.712343 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:34.716260 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:34.716270 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:34.716278 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:34.716322 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:34.719997 (+ 3675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:34.720742 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:34.720749 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:34.720753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:34.720908 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:34.721330 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:34.721334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:34.721338 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:34.721703 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:34.721872 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:34.722626 (+ 754us) server_negotiation.cc:299] Negotiation successful
0928 08:44:34.722793 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":311,"threads_started":1}
I20260928 08:44:34.724205 6744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:34.703431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.97.1:39577)
0928 08:44:34.704442 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:34.704506 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:34.707167 (+ 2661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:34.708038 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:34.708046 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:34.708565 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:34.709148 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:34.709167 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:34.710249 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:34.710262 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:34.711533 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:34.711546 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:34.712438 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:34.712448 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:34.712691 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:34.713431 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:34.713460 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:34.716034 (+ 2574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:34.720151 (+ 4117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:34.720158 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:34.720171 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:34.720600 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:34.721024 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:34.721027 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:34.721029 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:34.721195 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:34.721843 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:34.721849 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:34.722194 (+ 345us) client_negotiation.cc:769] Sending connection context
0928 08:44:34.722478 (+ 284us) client_negotiation.cc:240] Negotiation successful
0928 08:44:34.722877 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":377,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
I20260928 08:44:34.725925 6741 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46083
I20260928 08:44:34.726221 6741 heartbeater.cc:464] Registering TS with master...
I20260928 08:44:34.726984 6741 heartbeater.cc:511] Master 127.0.97.62:46083 requested a full tablet report, sending...
I20260928 08:44:34.729143 6552 ts_manager.cc:194] Registered new tserver with Master: fb125058c6c34aae8dff092e2f334d3b (127.0.97.1:46847)
I20260928 08:44:34.730542 6552 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39577
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260928 08:44:34.751329 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46083
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:35.082854 6747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:35.083158 6747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:35.083256 6747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:35.090626 6747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:35.090736 6747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:35.090867 6747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:44:35.101064 6747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46083
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:35.102485 6747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:35.103770 6747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:35.117149 6752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:35.117177 6753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:35.117110 6755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:35.118294 6747 server_base.cc:1034] running on GCE node
I20260928 08:44:35.118954 6747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:35.120126 6747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:35.121447 6747 hybrid_clock.cc:648] HybridClock initialized: now 1790585075121370 us; error 52 us; skew 500 ppm
I20260928 08:44:35.121778 6747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:35 dist-test-slave-n6xs krb5kdc[6507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585075, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:35.124667 6747 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260928 08:44:35.126056 6747 webserver.cc:460] Webserver started at http://127.0.97.2:34735/ using document root <none> and password file <none>
I20260928 08:44:35.126650 6747 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:35.126746 6747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:35.127038 6747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:35.129863 6747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-1/data/instance:
uuid: "1137082ae436457c84ba8100f189c4ec"
format_stamp: "Formatted at 2026-09-28 08:44:35 on dist-test-slave-n6xs"
server_key: "0cfe32ca52337c930027c9615ea2c7a3"
server_key_iv: "eff70e0cdcad695bb88cc7c40d30ed21"
server_key_version: "encryptionkey@0"
I20260928 08:44:35.130586 6747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-1/wal/instance:
uuid: "1137082ae436457c84ba8100f189c4ec"
format_stamp: "Formatted at 2026-09-28 08:44:35 on dist-test-slave-n6xs"
server_key: "0cfe32ca52337c930027c9615ea2c7a3"
server_key_iv: "eff70e0cdcad695bb88cc7c40d30ed21"
server_key_version: "encryptionkey@0"
I20260928 08:44:35.135421 6747 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 08:44:35.138720 6762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:35.140233 6747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:44:35.140399 6747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-1/wal
uuid: "1137082ae436457c84ba8100f189c4ec"
format_stamp: "Formatted at 2026-09-28 08:44:35 on dist-test-slave-n6xs"
server_key: "0cfe32ca52337c930027c9615ea2c7a3"
server_key_iv: "eff70e0cdcad695bb88cc7c40d30ed21"
server_key_version: "encryptionkey@0"
I20260928 08:44:35.140580 6747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:35.161567 6747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:35.165611 6747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:35.165906 6747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:35.166806 6747 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:44:35.168255 6747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:44:35.168361 6747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:35.168464 6747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:44:35.168532 6747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:35.201609 6747 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42561
I20260928 08:44:35.201678 6874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42561 every 8 connection(s)
I20260928 08:44:35.202826 6747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:44:35.211328 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 6747
I20260928 08:44:35.211515 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:44:35.212076 388 external_mini_cluster.cc:1428] Setting key 26d418e0781956b92a0de34b7488ed89
Sep 28 08:44:35 dist-test-slave-n6xs krb5kdc[6507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585075, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 08:44:35.222713 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:35.205276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57615 (local address 127.0.97.62:46083)
0928 08:44:35.205442 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:35.205446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:35.206555 (+ 1109us) server_negotiation.cc:407] Connection header received
0928 08:44:35.208353 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:35.208358 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:35.208441 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:35.208544 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:35.210092 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:35.210774 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:35.213027 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:35.213481 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:35.213581 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:35.217290 (+ 3709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:35.217294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:35.217297 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:35.217326 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:35.219899 (+ 2573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:35.220589 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:35.220594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:35.220598 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:35.220722 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:35.221131 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:35.221135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:35.221139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:35.221380 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:35.221530 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:35.222163 (+ 633us) server_negotiation.cc:299] Negotiation successful
0928 08:44:35.222372 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 08:44:35.223629 6877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:35.205742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.97.2:57615)
0928 08:44:35.206389 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:35.206424 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:35.208085 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:35.208804 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:35.208817 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:35.209327 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:35.209922 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:35.209948 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:35.210919 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:35.210924 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:35.212866 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:35.212880 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:35.213721 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:35.213731 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:35.214036 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:35.214722 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:35.214752 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:35.217075 (+ 2323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:35.220035 (+ 2960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:35.220043 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:35.220059 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:35.220468 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:35.220828 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:35.220834 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:35.220838 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:35.221020 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:35.221507 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:35.221514 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:35.221826 (+ 312us) client_negotiation.cc:769] Sending connection context
0928 08:44:35.222070 (+ 244us) client_negotiation.cc:240] Negotiation successful
0928 08:44:35.222574 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":258,"threads_started":1}
I20260928 08:44:35.225181 6875 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46083
I20260928 08:44:35.225529 6875 heartbeater.cc:464] Registering TS with master...
I20260928 08:44:35.226255 6875 heartbeater.cc:511] Master 127.0.97.62:46083 requested a full tablet report, sending...
I20260928 08:44:35.227941 6552 ts_manager.cc:194] Registered new tserver with Master: 1137082ae436457c84ba8100f189c4ec (127.0.97.2:42561)
I20260928 08:44:35.228863 6552 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:57615
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260928 08:44:35.259301 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46083
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:35.579303 6880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:35.579566 6880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:35.579659 6880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:35.586830 6880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:35.586937 6880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:35.587054 6880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:44:35.597004 6880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46083
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:35.598233 6880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:35.599431 6880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:35.612560 6885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:35.612596 6886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:35.612530 6888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:35.613358 6880 server_base.cc:1034] running on GCE node
I20260928 08:44:35.614014 6880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:35.615022 6880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:35.616276 6880 hybrid_clock.cc:648] HybridClock initialized: now 1790585075616219 us; error 39 us; skew 500 ppm
I20260928 08:44:35.616632 6880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:35 dist-test-slave-n6xs krb5kdc[6507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585075, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:35.619961 6880 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260928 08:44:35.621316 6880 webserver.cc:460] Webserver started at http://127.0.97.3:46489/ using document root <none> and password file <none>
I20260928 08:44:35.622008 6880 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:35.622150 6880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:35.622433 6880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:35.625267 6880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-2/data/instance:
uuid: "be8863ce64e24035aaeed4bf282343b3"
format_stamp: "Formatted at 2026-09-28 08:44:35 on dist-test-slave-n6xs"
server_key: "d7fd1951a7c8aebeb5be9678e5e58f80"
server_key_iv: "c5b9e6257c2822b706894c7a92215875"
server_key_version: "encryptionkey@0"
I20260928 08:44:35.626008 6880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-2/wal/instance:
uuid: "be8863ce64e24035aaeed4bf282343b3"
format_stamp: "Formatted at 2026-09-28 08:44:35 on dist-test-slave-n6xs"
server_key: "d7fd1951a7c8aebeb5be9678e5e58f80"
server_key_iv: "c5b9e6257c2822b706894c7a92215875"
server_key_version: "encryptionkey@0"
I20260928 08:44:35.631316 6880 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 08:44:35.634985 6895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:35.636323 6880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:44:35.636490 6880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-2/wal
uuid: "be8863ce64e24035aaeed4bf282343b3"
format_stamp: "Formatted at 2026-09-28 08:44:35 on dist-test-slave-n6xs"
server_key: "d7fd1951a7c8aebeb5be9678e5e58f80"
server_key_iv: "c5b9e6257c2822b706894c7a92215875"
server_key_version: "encryptionkey@0"
I20260928 08:44:35.636677 6880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:35.659740 6880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:35.663678 6880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:35.663988 6880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:35.664927 6880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:44:35.666178 6880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:44:35.666247 6880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:35.666420 6880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:44:35.666474 6880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:35.698941 6880 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41011
I20260928 08:44:35.699029 7007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41011 every 8 connection(s)
I20260928 08:44:35.700265 6880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:44:35.704617 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 6880
I20260928 08:44:35.704818 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:44:35.705217 388 external_mini_cluster.cc:1428] Setting key fdd7337b8de284949f94bc52cfcfa5aa
Sep 28 08:44:35 dist-test-slave-n6xs krb5kdc[6507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585075, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 08:44:35.720067 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:35.702965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45447 (local address 127.0.97.62:46083)
0928 08:44:35.703161 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:35.703167 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:35.704400 (+ 1233us) server_negotiation.cc:407] Connection header received
0928 08:44:35.706436 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:35.706443 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:35.706521 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:35.706608 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:35.708412 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:35.709174 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:35.710827 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:35.711203 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:35.711295 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:35.714543 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:35.714548 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:35.714551 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:35.714586 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:35.717040 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:35.717739 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:35.717743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:35.717745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:35.717929 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:35.718309 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:35.718313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:35.718316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:35.718582 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:35.718740 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:35.719479 (+ 739us) server_negotiation.cc:299] Negotiation successful
0928 08:44:35.719683 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 08:44:35.720978 7010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:35.703452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.97.3:45447)
0928 08:44:35.704200 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:35.704253 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:35.706156 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:35.706890 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:35.706902 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:35.707509 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:35.708270 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:35.708289 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:35.709300 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:35.709304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:35.710703 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:35.710715 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:35.711377 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:35.711381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:35.711569 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:35.712269 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:35.712289 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:35.714374 (+ 2085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:35.717169 (+ 2795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:35.717175 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:35.717186 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:35.717621 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:35.718026 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:35.718028 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:35.718031 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:35.718184 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:35.718718 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:35.718725 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:35.719119 (+ 394us) client_negotiation.cc:769] Sending connection context
0928 08:44:35.719379 (+ 260us) client_negotiation.cc:240] Negotiation successful
0928 08:44:35.719896 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":323,"threads_started":1}
I20260928 08:44:35.722405 7008 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46083
I20260928 08:44:35.722734 7008 heartbeater.cc:464] Registering TS with master...
I20260928 08:44:35.723470 7008 heartbeater.cc:511] Master 127.0.97.62:46083 requested a full tablet report, sending...
I20260928 08:44:35.725100 6552 ts_manager.cc:194] Registered new tserver with Master: be8863ce64e24035aaeed4bf282343b3 (127.0.97.3:41011)
I20260928 08:44:35.725946 6552 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45447
I20260928 08:44:35.732551 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:44:35.733282 6741 heartbeater.cc:503] Master 127.0.97.62:46083 was elected leader, sending a full tablet report...
WARNING: no policy specified for oryx/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/oryx_127.0.97.61.keytab.
Entry for principal oryx/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/oryx_127.0.97.61.keytab.
I20260928 08:44:35.780388 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:36547
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46083,127.0.97.61:36547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:36.103180 7013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:36.103561 7013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:36.103701 7013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:36.113963 7013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:44:36.114069 7013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:36.114115 7013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:36.114171 7013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:44:36.114216 7013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:44:36.123837 7013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46083,127.0.97.61:36547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/oryx_127.0.97.61.keytab
--principal=oryx/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:36547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
W20260928 08:44:36.124140 7013 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.97.62:46083,127.0.97.61:36547'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 08:44:36.125109 7013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:36.126327 7013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:36.137897 7018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:36.137962 7019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:36.137838 7021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:36.139025 7013 server_base.cc:1034] running on GCE node
I20260928 08:44:36.139763 7013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:36.141316 7013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:36.142553 7013 hybrid_clock.cc:648] HybridClock initialized: now 1790585076142469 us; error 33 us; skew 500 ppm
I20260928 08:44:36.142838 7013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:36 dist-test-slave-n6xs krb5kdc[6507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585076, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:36.146111 7013 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260928 08:44:36.147491 7013 webserver.cc:460] Webserver started at http://127.0.97.61:38463/ using document root <none> and password file <none>
I20260928 08:44:36.148267 7013 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:36.148406 7013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:36.148746 7013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:36.151595 7013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-1/data/instance:
uuid: "17003f6ce92c4415a110af3b597c8e4a"
format_stamp: "Formatted at 2026-09-28 08:44:36 on dist-test-slave-n6xs"
server_key: "25c8dc3a1dff5a3692b4dcea310d5425"
server_key_iv: "b771694d543f9fa2349921108c262a64"
server_key_version: "encryptionkey@0"
I20260928 08:44:36.152405 7013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-1/wal/instance:
uuid: "17003f6ce92c4415a110af3b597c8e4a"
format_stamp: "Formatted at 2026-09-28 08:44:36 on dist-test-slave-n6xs"
server_key: "25c8dc3a1dff5a3692b4dcea310d5425"
server_key_iv: "b771694d543f9fa2349921108c262a64"
server_key_version: "encryptionkey@0"
I20260928 08:44:36.157639 7013 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 08:44:36.160961 7028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:36.162335 7013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:44:36.162529 7013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-1/wal
uuid: "17003f6ce92c4415a110af3b597c8e4a"
format_stamp: "Formatted at 2026-09-28 08:44:36 on dist-test-slave-n6xs"
server_key: "25c8dc3a1dff5a3692b4dcea310d5425"
server_key_iv: "b771694d543f9fa2349921108c262a64"
server_key_version: "encryptionkey@0"
I20260928 08:44:36.162740 7013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:36.175662 7013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:36.179615 7013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:36.179917 7013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:36.197971 7013 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:36547
I20260928 08:44:36.197996 7079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:36547 every 8 connection(s)
I20260928 08:44:36.199318 7013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-1/data/info.pb
I20260928 08:44:36.202920 7080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:36.209708 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 7013
I20260928 08:44:36.209260 7080 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:44:36 dist-test-slave-n6xs krb5kdc[6507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585076, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 28 08:44:36 dist-test-slave-n6xs krb5kdc[6507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260928 08:44:36.231379 6875 heartbeater.cc:503] Master 127.0.97.62:46083 was elected leader, sending a full tablet report...
I20260928 08:44:36.236297 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:36.212405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35926 (local address 127.0.97.62:46083)
0928 08:44:36.212628 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:36.212634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:36.213498 (+ 864us) server_negotiation.cc:407] Connection header received
0928 08:44:36.214796 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:36.214801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:36.214881 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:36.214973 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:36.216775 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.217914 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.220084 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.220729 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.220863 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:36.229465 (+ 8602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:36.229471 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:36.229475 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:36.229505 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:36.232686 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:36.233586 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:36.233592 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:36.233597 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:36.233735 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:36.234264 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:36.234269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:36.234271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:36.234697 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:36.234800 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:36.235674 (+ 874us) server_negotiation.cc:299] Negotiation successful
0928 08:44:36.235928 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260928 08:44:36.237573 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:36.212012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48768 (local address 127.0.97.61:36547)
0928 08:44:36.212793 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:36.212802 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:36.212838 (+ 36us) server_negotiation.cc:407] Connection header received
0928 08:44:36.214483 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:36.214502 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:36.214755 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:36.214959 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:36.216802 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.218339 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.220084 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.220894 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.221076 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:36.230108 (+ 9032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:36.230116 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:36.230128 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:36.230156 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:36.233251 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:36.234020 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:36.234027 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:36.234030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:36.234170 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:36.234599 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:36.234602 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:36.234604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:36.235297 (+ 693us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:36.235597 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:36.236027 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 08:44:36.236220 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":579,"spinlock_wait_cycles":20224,"thread_start_us":360,"threads_started":1}
I20260928 08:44:36.237607 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:36.212508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.0.1:35926)
0928 08:44:36.213310 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:36.213369 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:36.214537 (+ 1168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:36.215076 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:36.215082 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:36.216012 (+ 930us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:36.216578 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:36.216605 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.218080 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.218084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:36.219942 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:36.219957 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.220992 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.221001 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:36.221180 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:36.226220 (+ 5040us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:36.226252 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:36.229252 (+ 3000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:36.232821 (+ 3569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:36.232845 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:36.232859 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:36.233468 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:36.233884 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:36.233887 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:36.233889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:36.234110 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:36.234811 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:36.234818 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:36.235268 (+ 450us) client_negotiation.cc:769] Sending connection context
0928 08:44:36.235565 (+ 297us) client_negotiation.cc:240] Negotiation successful
0928 08:44:36.236113 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"mutex_wait_us":59,"thread_start_us":411,"threads_started":1}
I20260928 08:44:36.239286 7080 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:44:36 dist-test-slave-n6xs krb5kdc[6507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585076, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260928 08:44:36.251724 7084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:36.240149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36547 (local address 127.0.0.1:48784)
0928 08:44:36.240471 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:36.240490 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:36.240597 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:36.240899 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:36.240902 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:36.241199 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:36.241332 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:36.241340 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.242427 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.242431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:36.243612 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:36.243624 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.244368 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.244373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:36.244524 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:36.245077 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:36.245093 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:36.247103 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:36.249863 (+ 2760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:36.249867 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:36.249869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:36.250200 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:36.250506 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:36.250508 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:36.250510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:36.250622 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:36.251063 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:36.251066 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:36.251182 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 08:44:36.251290 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 08:44:36.251479 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210}
I20260928 08:44:36.251895 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:36.240059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48784 (local address 127.0.97.61:36547)
0928 08:44:36.240471 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:36.240477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:36.240565 (+ 88us) server_negotiation.cc:407] Connection header received
0928 08:44:36.240673 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:36.240676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:36.240728 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:36.240799 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:36.241467 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.242296 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.243731 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.244238 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.244335 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:36.247262 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:36.247267 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:36.247271 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:36.247296 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:36.249739 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:36.250294 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:36.250298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:36.250300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:36.250419 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:36.250705 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:36.250708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:36.250710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:36.250947 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:36.251066 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:36.251373 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 08:44:36.251573 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":295}
I20260928 08:44:36.256390 7080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a: Bootstrap starting.
I20260928 08:44:36.259291 7080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:36.260322 7080 log.cc:824] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a: Log is configured to *not* fsync() on all Append() calls
I20260928 08:44:36.262727 7080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a: No bootstrap required, opened a new log
I20260928 08:44:36.269919 7080 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } }
I20260928 08:44:36.270308 7080 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:36.270409 7080 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17003f6ce92c4415a110af3b597c8e4a, State: Initialized, Role: FOLLOWER
I20260928 08:44:36.270959 7080 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } }
I20260928 08:44:36.271726 7086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } } }
I20260928 08:44:36.272146 7086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:44:36.272768 7080 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [sys.catalog]: configured and running, proceeding with master startup.
W20260928 08:44:36.282025 7097 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:44:36.282166 7097 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:44:36.283589 7013 master_runner.cc:428] Detected that this master 17003f6ce92c4415a110af3b597c8e4a is joining an existing cluster
I20260928 08:44:36.283699 7013 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.61:36547
I20260928 08:44:36.304692 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:36.287274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48796 (local address 127.0.97.61:36547)
0928 08:44:36.287507 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:36.287512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:36.288925 (+ 1413us) server_negotiation.cc:407] Connection header received
0928 08:44:36.289069 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:36.289075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:36.289151 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:36.289269 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:36.290662 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.291995 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.294094 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.294983 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.295112 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:36.297801 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:36.297829 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:36.297835 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:36.297870 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:36.301822 (+ 3952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:36.302634 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:36.302642 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:36.302646 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:36.302770 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:36.303217 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:36.303221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:36.303223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:36.303565 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:36.303663 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:36.304181 (+ 518us) server_negotiation.cc:299] Negotiation successful
0928 08:44:36.304410 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":50}
I20260928 08:44:36.305425 7104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:36.287924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36547 (local address 127.0.0.1:48796)
0928 08:44:36.288817 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:36.288836 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:36.288960 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:36.289321 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:36.289331 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:36.290143 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:36.290484 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:36.290497 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.292241 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.292246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:36.293931 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:36.293945 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.295188 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.295195 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:36.295435 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:36.296279 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:36.296305 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:36.297614 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:36.301993 (+ 4379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:36.302000 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:36.302007 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:36.302529 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:36.302881 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:36.302892 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:36.302895 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:36.303065 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:36.303729 (+ 664us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:36.303732 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:36.303923 (+ 191us) client_negotiation.cc:769] Sending connection context
0928 08:44:36.304898 (+ 975us) client_negotiation.cc:240] Negotiation successful
0928 08:44:36.305171 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"mutex_wait_us":32,"thread_start_us":546,"threads_started":1}
I20260928 08:44:36.305830 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:36.287274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35938 (local address 127.0.97.62:46083)
0928 08:44:36.287461 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:36.287467 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:36.288328 (+ 861us) server_negotiation.cc:407] Connection header received
0928 08:44:36.288568 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:36.288587 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:36.288660 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:36.288747 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:36.290068 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.290994 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.293156 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.293899 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.294007 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:36.297942 (+ 3935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:36.297950 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:36.297958 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:36.298000 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:36.302318 (+ 4318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:36.303197 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:36.303204 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:36.303207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:36.303355 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:36.303837 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:36.303841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:36.303844 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:36.304205 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:36.304345 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:36.305062 (+ 717us) server_negotiation.cc:299] Negotiation successful
0928 08:44:36.305310 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 08:44:36.307085 7103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:36.287373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.0.1:35938)
0928 08:44:36.288183 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:36.288217 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:36.288392 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:36.288926 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:36.288930 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:36.289597 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:36.289926 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:36.289937 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.291145 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.291151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:36.292984 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:36.292999 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.294305 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.294315 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:36.295499 (+ 1184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:36.296411 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:36.296436 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:36.297737 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:36.302451 (+ 4714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:36.302456 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:36.302462 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:36.303031 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:36.303473 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:36.303479 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:36.303482 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:36.303672 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:36.304349 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:36.304357 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:36.304598 (+ 241us) client_negotiation.cc:769] Sending connection context
0928 08:44:36.306108 (+ 1510us) client_negotiation.cc:240] Negotiation successful
0928 08:44:36.306466 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":120,"thread_start_us":506,"threads_started":1}
WARNING: no policy specified for oryx/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.60@KRBTEST.COM" created.
Sep 28 08:44:36 dist-test-slave-n6xs krb5kdc[6507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585074, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260928 08:44:36.332562 7106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:36.314266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36547 (local address 127.0.0.1:48804)
0928 08:44:36.318073 (+ 3807us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:36.318105 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:36.318294 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:36.318772 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:36.318779 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:36.319436 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:36.319631 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:36.319640 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.320897 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.320902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:36.321855 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:36.321865 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.322664 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.322670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:36.322847 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:36.323468 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:36.323485 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:36.325527 (+ 2042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:36.329227 (+ 3700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:36.329242 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:36.329252 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:36.329826 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:36.330283 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:36.330290 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:36.330297 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:36.330493 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:36.331154 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:36.331166 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:36.331397 (+ 231us) client_negotiation.cc:769] Sending connection context
0928 08:44:36.331615 (+ 218us) client_negotiation.cc:240] Negotiation successful
0928 08:44:36.332008 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3632,"thread_start_us":3504,"threads_started":1}
I20260928 08:44:36.333645 7083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:36.314329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48804 (local address 127.0.97.61:36547)
0928 08:44:36.314557 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:36.314565 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:36.318196 (+ 3631us) server_negotiation.cc:407] Connection header received
0928 08:44:36.318542 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:36.318547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:36.318628 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:36.318735 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:36.319777 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.320762 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.321999 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.322491 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.322651 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:36.325706 (+ 3055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:36.325711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:36.325715 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:36.325741 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:36.329094 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:36.329967 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:36.329974 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:36.329978 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:36.330150 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:36.330624 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:36.330629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:36.330633 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:36.331007 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:36.331164 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:36.333016 (+ 1852us) server_negotiation.cc:299] Negotiation successful
0928 08:44:36.333252 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 08:44:36.335685 6552 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.61:36547: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.61"
port: 36547
}
attrs {
promote: true
}
}
dest_uuid: "0c813b2df2b645b18971acadaa4ca63a"
cas_config_opid_index: -1
I20260928 08:44:36.336970 6552 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: true } }
I20260928 08:44:36.345625 7055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:36.345059 7110 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 17003f6ce92c4415a110af3b597c8e4a (127.0.97.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: true } } }
I20260928 08:44:36.349280 7110 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.61:36547
I20260928 08:44:36.349494 7055 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [term 1 FOLLOWER]: Refusing update from remote peer 0c813b2df2b645b18971acadaa4ca63a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 08:44:36.350028 7013 master.cc:537] Master@127.0.97.61:36547 shutting down...
I20260928 08:44:36.352118 7109 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:44:36.352367 7111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0c813b2df2b645b18971acadaa4ca63a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: true } } }
I20260928 08:44:36.352871 7111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [sys.catalog]: This master's current role is: LEADER
W20260928 08:44:36.356184 6538 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/oryx_127.0.97.60.keytab.
Entry for principal oryx/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/oryx_127.0.97.60.keytab.
W20260928 08:44:36.358036 6538 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a -> Peer 17003f6ce92c4415a110af3b597c8e4a (127.0.97.61:36547): Couldn't send request to peer 17003f6ce92c4415a110af3b597c8e4a. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 08:44:36.359464 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:37393
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46083,127.0.97.61:36547,127.0.97.60:37393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:44:36.362479 7013 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:44:36.362921 7013 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:36.363128 7013 tablet_replica.cc:330] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a: stopping tablet replica
I20260928 08:44:36.369729 7013 master.cc:552] Master@127.0.97.61:36547 shutdown complete.
I20260928 08:44:36.369819 7013 master_runner.cc:305] Clearing existing system tablet
I20260928 08:44:36.370237 7013 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:36.370357 7013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:36.372977 7013 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 08:44:36.374671 7116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:36.375022 7013 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:36.375327 7013 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 08:44:36.375425 7013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-1/wal
uuid: "17003f6ce92c4415a110af3b597c8e4a"
format_stamp: "Formatted at 2026-09-28 08:44:36 on dist-test-slave-n6xs"
server_key: "25c8dc3a1dff5a3692b4dcea310d5425"
server_key_iv: "b771694d543f9fa2349921108c262a64"
server_key_version: "encryptionkey@0"
I20260928 08:44:36.378993 7013 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:36.383971 7013 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 08:44:36.384195 7013 log.cc:1196] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 08:44:36.384675 7013 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a: Deleting consensus metadata
I20260928 08:44:36.384986 7013 master_runner.cc:315] Copying system tablet from 127.0.97.62:46083
I20260928 08:44:36.387439 7013 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:46083
I20260928 08:44:36.401064 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:36.388259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35948 (local address 127.0.97.62:46083)
0928 08:44:36.388520 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:36.388529 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:36.389094 (+ 565us) server_negotiation.cc:407] Connection header received
0928 08:44:36.389341 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:36.389347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:36.389437 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:36.389573 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:36.390492 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.391416 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.392810 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.393277 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.393424 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:36.395250 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:36.395257 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:36.395264 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:36.395309 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:36.398312 (+ 3003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:36.398982 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:36.398990 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:36.398994 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:36.399122 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:36.399516 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:36.399522 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:36.399525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:36.399909 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:36.400068 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:36.400465 (+ 397us) server_negotiation.cc:299] Negotiation successful
0928 08:44:36.400669 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 08:44:36.401265 7122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:36.388156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.0.1:35948)
0928 08:44:36.388975 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:36.388992 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:36.389165 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:36.389604 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:36.389609 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:36.390088 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:36.390315 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:36.390325 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.391567 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.391574 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:36.392655 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:36.392667 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.393459 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.393465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:36.393640 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:36.394319 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:36.394338 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:36.395028 (+ 690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:36.398457 (+ 3429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:36.398465 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:36.398469 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:36.398857 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:36.399247 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:36.399254 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:36.399258 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:36.399398 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:36.400024 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:36.400029 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:36.400193 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 08:44:36.400357 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 08:44:36.400669 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":548,"threads_started":1}
I20260928 08:44:36.402370 6572 tablet_copy_service.cc:140] P 0c813b2df2b645b18971acadaa4ca63a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 17003f6ce92c4415a110af3b597c8e4a ({username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:35948)
I20260928 08:44:36.402649 6572 tablet_copy_service.cc:161] P 0c813b2df2b645b18971acadaa4ca63a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 17003f6ce92c4415a110af3b597c8e4a at {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:35948: session id = 17003f6ce92c4415a110af3b597c8e4a-00000000000000000000000000000000
I20260928 08:44:36.405833 6572 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 08:44:36.407763 7013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 08:44:36.412806 7013 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a: tablet copy: Starting download of 0 data blocks...
I20260928 08:44:36.413113 7013 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a: tablet copy: Starting download of 1 WAL segments...
I20260928 08:44:36.415619 7013 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 08:44:36.418860 6572 tablet_copy_service.cc:342] P 0c813b2df2b645b18971acadaa4ca63a: Request end of tablet copy session 17003f6ce92c4415a110af3b597c8e4a-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.61@KRBTEST.COM'} at 127.0.0.1:35948
I20260928 08:44:36.419050 6572 tablet_copy_service.cc:433] P 0c813b2df2b645b18971acadaa4ca63a: ending tablet copy session 17003f6ce92c4415a110af3b597c8e4a-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 17003f6ce92c4415a110af3b597c8e4a
I20260928 08:44:36.430074 7013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:36.432902 7123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:36.438978 7124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:36.443656 7126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:36.444123 7013 server_base.cc:1034] running on GCE node
I20260928 08:44:36.444446 7013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:36.445276 7013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:36.446520 7013 hybrid_clock.cc:648] HybridClock initialized: now 1790585076446455 us; error 40 us; skew 500 ppm
I20260928 08:44:36.446765 7013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:36 dist-test-slave-n6xs krb5kdc[6507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585076, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:36.448832 7013 init.cc:377] Logged in from keytab as oryx/127.0.97.61@KRBTEST.COM (short username oryx)
I20260928 08:44:36.450069 7013 webserver.cc:460] Webserver started at http://127.0.97.61:44459/ using document root <none> and password file <none>
I20260928 08:44:36.450392 7013 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:36.450502 7013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:36.453044 7013 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260928 08:44:36.454874 7133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:36.455272 7013 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260928 08:44:36.455425 7013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-1/wal
uuid: "17003f6ce92c4415a110af3b597c8e4a"
format_stamp: "Formatted at 2026-09-28 08:44:36 on dist-test-slave-n6xs"
server_key: "25c8dc3a1dff5a3692b4dcea310d5425"
server_key_iv: "b771694d543f9fa2349921108c262a64"
server_key_version: "encryptionkey@0"
I20260928 08:44:36.455603 7013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:36.475087 7013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:36.478638 7013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:36.500520 7013 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:36547
I20260928 08:44:36.500561 7184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:36547 every 8 connection(s)
I20260928 08:44:36.501477 7013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-1/data/info.pb
I20260928 08:44:36.508785 7185 sys_catalog.cc:263] Verifying existing consensus state
I20260928 08:44:36.515298 7185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a: Bootstrap starting.
I20260928 08:44:36.533120 7185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:44:36.533648 7185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a: Bootstrap complete.
I20260928 08:44:36.534675 7185 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: true } }
I20260928 08:44:36.534922 7185 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 17003f6ce92c4415a110af3b597c8e4a, State: Initialized, Role: LEARNER
I20260928 08:44:36.535151 7185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: true } }
I20260928 08:44:36.536264 7188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: true } } }
I20260928 08:44:36.536537 7185 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:44:36.536674 7188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [sys.catalog]: This master's current role is: LEARNER
I20260928 08:44:36.552281 7199 catalog_manager.cc:1259] Loaded cluster ID: cd79b6032ebe422ab25f6fc91d390632
I20260928 08:44:36.552402 7199 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a: loading cluster ID for follower catalog manager: success
I20260928 08:44:36.556198 7199 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a: acquiring CA information for follower catalog manager: success
I20260928 08:44:36.558808 7199 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 08:44:36.728628 7008 heartbeater.cc:503] Master 127.0.97.62:46083 was elected leader, sending a full tablet report...
W20260928 08:44:36.769587 7114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:36.769987 7114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:36.770125 7114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:36.781539 7114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:44:36.781699 7114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:36.781777 7114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:36.781862 7114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:44:36.781925 7114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:44:36.792366 7114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46083,127.0.97.61:36547,127.0.97.60:37393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/oryx_127.0.97.60.keytab
--principal=oryx/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:37393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:36.793682 7114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:36.794962 7114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:36.806696 7207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:36.806655 7205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:36.806906 7204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:36.809383 7114 server_base.cc:1034] running on GCE node
I20260928 08:44:36.810199 7114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:36.811641 7114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:36.812897 7114 hybrid_clock.cc:648] HybridClock initialized: now 1790585076812851 us; error 53 us; skew 500 ppm
I20260928 08:44:36.813252 7114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:36 dist-test-slave-n6xs krb5kdc[6507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585076, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:36.816768 7114 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260928 08:44:36.818197 7114 webserver.cc:460] Webserver started at http://127.0.97.60:46029/ using document root <none> and password file <none>
I20260928 08:44:36.818861 7114 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:36.818957 7114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:36.819265 7114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:36.822655 7114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-2/data/instance:
uuid: "a63285bae4b64ecca10248bb821d965f"
format_stamp: "Formatted at 2026-09-28 08:44:36 on dist-test-slave-n6xs"
server_key: "0282e586ebcd48df2158fffc111f0ba6"
server_key_iv: "0bbca4511b46d5b4f8e0657f40c1d882"
server_key_version: "encryptionkey@0"
I20260928 08:44:36.823424 7114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-2/wal/instance:
uuid: "a63285bae4b64ecca10248bb821d965f"
format_stamp: "Formatted at 2026-09-28 08:44:36 on dist-test-slave-n6xs"
server_key: "0282e586ebcd48df2158fffc111f0ba6"
server_key_iv: "0bbca4511b46d5b4f8e0657f40c1d882"
server_key_version: "encryptionkey@0"
I20260928 08:44:36.828727 7114 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260928 08:44:36.832329 7214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:36.833745 7114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 08:44:36.833953 7114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-2/wal
uuid: "a63285bae4b64ecca10248bb821d965f"
format_stamp: "Formatted at 2026-09-28 08:44:36 on dist-test-slave-n6xs"
server_key: "0282e586ebcd48df2158fffc111f0ba6"
server_key_iv: "0bbca4511b46d5b4f8e0657f40c1d882"
server_key_version: "encryptionkey@0"
I20260928 08:44:36.834167 7114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:36.878080 7114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:36.883179 7114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:36.883486 7114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:36.904078 7114 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:37393
I20260928 08:44:36.904127 7265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:37393 every 8 connection(s)
I20260928 08:44:36.905454 7114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-2/data/info.pb
I20260928 08:44:36.909572 7266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:36.911001 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 7114
I20260928 08:44:36.918414 7266 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:44:36 dist-test-slave-n6xs krb5kdc[6507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 28 08:44:36 dist-test-slave-n6xs krb5kdc[6507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585076, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 08:44:36.938717 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:36.913280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40174 (local address 127.0.97.60:37393)
0928 08:44:36.914137 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:36.914148 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:36.914189 (+ 41us) server_negotiation.cc:407] Connection header received
0928 08:44:36.916086 (+ 1897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:36.916109 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:36.916512 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:36.916922 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:36.918374 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.919745 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.921291 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.922364 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.922540 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:36.927745 (+ 5205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:36.927756 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:36.927772 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:36.927866 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:36.931275 (+ 3409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:36.932112 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:36.932121 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:36.932131 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:36.932302 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:36.932785 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:36.932788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:36.932790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:36.933513 (+ 723us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:36.935360 (+ 1847us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:36.935748 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 08:44:36.936483 (+ 735us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":371,"threads_started":1}
I20260928 08:44:36.940491 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:36.920778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.0.1:35952)
0928 08:44:36.921522 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:36.921572 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:36.921707 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:36.922502 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:36.922509 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:36.923839 (+ 1330us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:36.924157 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:36.924169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.925249 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.925256 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:36.926628 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:36.926641 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.927700 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.927710 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:36.927993 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:36.928700 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:36.928722 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:36.931624 (+ 2902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:36.936187 (+ 4563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:36.936204 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:36.936214 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:36.936988 (+ 774us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:36.937544 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:36.937552 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:36.937558 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:36.937740 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:36.938617 (+ 877us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:36.938629 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:36.938917 (+ 288us) client_negotiation.cc:769] Sending connection context
0928 08:44:36.939155 (+ 238us) client_negotiation.cc:240] Negotiation successful
0928 08:44:36.939704 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":322,"threads_started":1}
I20260928 08:44:36.940804 7271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:36.921062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35952 (local address 127.0.97.62:46083)
0928 08:44:36.922113 (+ 1051us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:36.922119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:36.922137 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:44:36.922190 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:36.922195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:36.922270 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:36.922379 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:36.924326 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.925098 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.926776 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.927463 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.927605 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:36.931792 (+ 4187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:36.931796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:36.931799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:36.931879 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:36.935994 (+ 4115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:36.937182 (+ 1188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:36.937190 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:36.937196 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:36.937368 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:36.937929 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:36.937937 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:36.937944 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:36.938363 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:36.938534 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:36.940005 (+ 1471us) server_negotiation.cc:299] Negotiation successful
0928 08:44:36.940279 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":940,"thread_start_us":847,"threads_started":1}
I20260928 08:44:36.943256 7266 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:44:36.947746 7273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:36.934069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36547 (local address 127.0.0.1:48818)
0928 08:44:36.941558 (+ 7489us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:36.941573 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:36.941688 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:36.942083 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:36.942088 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:36.942116 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:36.942411 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:36.942422 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.944206 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.944212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:36.945702 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:36.945714 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.946883 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.946889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:36.947085 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:36.947142 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 08:44:36.947254 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 08:44:36.947388 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7358,"thread_start_us":7233,"threads_started":1}
I20260928 08:44:36.947760 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:36.934236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48818 (local address 127.0.97.61:36547)
0928 08:44:36.941109 (+ 6873us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:36.941118 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:36.941659 (+ 541us) server_negotiation.cc:407] Connection header received
0928 08:44:36.941795 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:36.941800 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:36.941900 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:36.942056 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:36.942591 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.944030 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.945864 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.946684 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.946837 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:36.947053 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:36.947279 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 08:44:36.947413 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6707,"thread_start_us":8133,"threads_started":1}
I20260928 08:44:36.949311 7160 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260928 08:44:36.949954 7188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c813b2df2b645b18971acadaa4ca63a. Latest consensus state: current_term: 1 leader_uuid: "0c813b2df2b645b18971acadaa4ca63a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: true } } }
I20260928 08:44:36.950212 7188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [sys.catalog]: This master's current role is: LEARNER
Sep 28 08:44:36 dist-test-slave-n6xs krb5kdc[6507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585076, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
I20260928 08:44:36.960758 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:36.944025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36547 (local address 127.0.0.1:48830)
0928 08:44:36.944212 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:36.944231 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:36.944330 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:36.948275 (+ 3945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:36.948282 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:36.948760 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:36.948931 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:36.948946 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.949977 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.949984 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:36.951037 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:36.951056 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.951728 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.951733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:36.951970 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:36.952529 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:36.952564 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:36.955161 (+ 2597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:36.958326 (+ 3165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:36.958331 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:36.958336 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:36.958722 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:36.959108 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:36.959113 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:36.959117 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:36.959248 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:36.959762 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:36.959767 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:36.959950 (+ 183us) client_negotiation.cc:769] Sending connection context
0928 08:44:36.960087 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 08:44:36.960391 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 08:44:36.960793 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:36.944151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48830 (local address 127.0.97.61:36547)
0928 08:44:36.948013 (+ 3862us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:36.948018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:36.948041 (+ 23us) server_negotiation.cc:407] Connection header received
0928 08:44:36.948079 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:36.948082 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:36.948135 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:36.948245 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:36.949072 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.949847 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.951180 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.951585 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.951715 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:36.955353 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:36.955358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:36.955361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:36.955443 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:36.958185 (+ 2742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:36.958852 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:36.958864 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:36.958867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:36.958992 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:36.959360 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:36.959366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:36.959369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:36.959648 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:36.959799 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:36.960192 (+ 393us) server_negotiation.cc:299] Negotiation successful
0928 08:44:36.960408 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3764,"thread_start_us":4519,"threads_started":1}
I20260928 08:44:36.961997 7266 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37393 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:44:36 dist-test-slave-n6xs krb5kdc[6507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585076, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260928 08:44:36.974495 7270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:36.962907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37393 (local address 127.0.0.1:40176)
0928 08:44:36.963130 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:36.963152 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:36.963304 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:36.963594 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:36.963598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:36.964063 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:36.964194 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:36.964202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.965130 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.965137 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:36.965943 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:36.965953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.966627 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.966632 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:36.966842 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:36.967428 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:36.967445 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:36.969478 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:36.972270 (+ 2792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:36.972273 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:36.972276 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:36.972617 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:36.973133 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:36.973136 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:36.973139 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:36.973289 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:36.973751 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:36.973754 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:36.973906 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 08:44:36.974027 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 08:44:36.974219 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260928 08:44:36.974579 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:36.962916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40176 (local address 127.0.97.60:37393)
0928 08:44:36.963130 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:36.963137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:36.963234 (+ 97us) server_negotiation.cc:407] Connection header received
0928 08:44:36.963419 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:36.963423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:36.963485 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:36.963582 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:36.964340 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.965009 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.966071 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:36.966480 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:36.966628 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:36.969667 (+ 3039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:36.969671 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:36.969674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:36.969702 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:36.972158 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:36.972714 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:36.972718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:36.972720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:36.972866 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:36.973391 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:36.973394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:36.973396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:36.973644 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:36.973748 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:36.974118 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 08:44:36.974317 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":84}
I20260928 08:44:36.979445 7266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f: Bootstrap starting.
I20260928 08:44:36.983194 7266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:36.984309 7266 log.cc:824] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f: Log is configured to *not* fsync() on all Append() calls
I20260928 08:44:36.987080 7266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f: No bootstrap required, opened a new log
I20260928 08:44:36.994621 7266 raft_consensus.cc:348] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37393 } }
I20260928 08:44:36.994966 7266 raft_consensus.cc:374] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:36.995098 7266 raft_consensus.cc:729] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a63285bae4b64ecca10248bb821d965f, State: Initialized, Role: FOLLOWER
I20260928 08:44:36.995702 7266 consensus_queue.cc:260] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37393 } }
I20260928 08:44:36.997164 7277 sys_catalog.cc:455] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37393 } } }
I20260928 08:44:36.997593 7277 sys_catalog.cc:458] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:44:36.998067 7266 sys_catalog.cc:564] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [sys.catalog]: configured and running, proceeding with master startup.
W20260928 08:44:37.010007 7288 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:44:37.010159 7288 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:44:37.014390 7114 master_runner.cc:428] Detected that this master a63285bae4b64ecca10248bb821d965f is joining an existing cluster
I20260928 08:44:37.014518 7114 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.97.60:37393
I20260928 08:44:37.044484 7271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:37.019144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35968 (local address 127.0.97.62:46083)
0928 08:44:37.019349 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:37.019358 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:37.027031 (+ 7673us) server_negotiation.cc:407] Connection header received
0928 08:44:37.027399 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:37.027408 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:37.027512 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:37.027597 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:37.029520 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.030219 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.031598 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.032219 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.032349 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:37.035437 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:37.035444 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:37.035449 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:37.035488 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:37.040131 (+ 4643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:37.041147 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:37.041153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:37.041156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:37.041315 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:37.042335 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:37.042345 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:37.042351 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:37.042729 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:37.042904 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:37.043878 (+ 974us) server_negotiation.cc:299] Negotiation successful
0928 08:44:37.044102 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 08:44:37.046322 7295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:37.020063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.0.1:35968)
0928 08:44:37.026885 (+ 6822us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:37.026907 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:37.027213 (+ 306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:37.027878 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:37.027883 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:37.029091 (+ 1208us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:37.029366 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:37.029374 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.030375 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.030380 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:37.031459 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:37.031470 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.032606 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.032612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:37.032784 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:37.033556 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:37.033980 (+ 424us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:37.035238 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:37.040290 (+ 5052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:37.040296 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:37.040300 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:37.041014 (+ 714us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:37.041446 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:37.041450 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:37.041453 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:37.042222 (+ 769us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:37.042968 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:37.042978 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:37.043242 (+ 264us) client_negotiation.cc:769] Sending connection context
0928 08:44:37.043419 (+ 177us) client_negotiation.cc:240] Negotiation successful
0928 08:44:37.045460 (+ 2041us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6635,"thread_start_us":6409,"threads_started":1}
I20260928 08:44:37.046355 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 6613
I20260928 08:44:37.050407 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:37.019638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48838 (local address 127.0.97.61:36547)
0928 08:44:37.019841 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:37.019847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:37.026329 (+ 6482us) server_negotiation.cc:407] Connection header received
0928 08:44:37.026938 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:37.026945 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:37.027029 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:37.027124 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:37.032264 (+ 5140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.033339 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.036936 (+ 3597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.037603 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.037693 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:37.040329 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:37.040336 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:37.040341 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:37.040381 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:37.044224 (+ 3843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:37.047678 (+ 3454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:37.047684 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:37.047688 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:37.047942 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:37.048421 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:37.048427 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:37.048430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:37.048783 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:37.048933 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:37.049532 (+ 599us) server_negotiation.cc:299] Negotiation successful
0928 08:44:37.050057 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 08:44:37.051735 7294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:37.019316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36547 (local address 127.0.0.1:48838)
0928 08:44:37.026207 (+ 6891us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:37.026228 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:37.026749 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:37.027388 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:37.027393 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:37.028208 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:37.032080 (+ 3872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:37.032093 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.034044 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.034053 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:37.036696 (+ 2643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:37.036709 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.038052 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.038064 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:37.038261 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:37.038887 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:37.038908 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:37.040105 (+ 1197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:37.044341 (+ 4236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:37.044345 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:37.044347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:37.047532 (+ 3185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:37.048098 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:37.048103 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:37.048106 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:37.048288 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:37.049011 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:37.049016 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:37.049241 (+ 225us) client_negotiation.cc:769] Sending connection context
0928 08:44:37.050909 (+ 1668us) client_negotiation.cc:240] Negotiation successful
0928 08:44:37.051258 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6750,"mutex_wait_us":2,"thread_start_us":6605,"threads_started":1}
I20260928 08:44:37.051895 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:37.019316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40192 (local address 127.0.97.60:37393)
0928 08:44:37.019575 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:37.019581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:37.032275 (+ 12694us) server_negotiation.cc:407] Connection header received
0928 08:44:37.032446 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:37.032450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:37.032510 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:37.032609 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:37.034537 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.035611 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.037447 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.038240 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.038376 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:37.041299 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:37.041304 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:37.041308 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:37.041340 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:37.046969 (+ 5629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:37.047850 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:37.047856 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:37.047859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:37.048005 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:37.048403 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:37.048410 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:37.048413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:37.048767 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:37.048933 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:37.049414 (+ 481us) server_negotiation.cc:299] Negotiation successful
0928 08:44:37.049663 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":118}
I20260928 08:44:37.057113 7296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:37.020084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37393 (local address 127.0.0.1:40192)
0928 08:44:37.032174 (+ 12090us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:37.032189 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:37.032328 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:37.032756 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:37.032764 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:37.033461 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:37.034355 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:37.034373 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.035776 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.035781 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:37.037310 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:37.037327 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.038680 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.038688 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:37.038894 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:37.039499 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:37.039526 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:37.040761 (+ 1235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:37.047104 (+ 6343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:37.047109 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:37.047112 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:37.047686 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:37.048145 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:37.048152 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:37.048155 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:37.048291 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:37.048996 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:37.049000 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:37.049180 (+ 180us) client_negotiation.cc:769] Sending connection context
0928 08:44:37.056496 (+ 7316us) client_negotiation.cc:240] Negotiation successful
0928 08:44:37.056728 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11976,"thread_start_us":11785,"threads_started":1}
Sep 28 08:44:37 dist-test-slave-n6xs krb5kdc[6507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585074, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260928 08:44:37.070722 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:37.053814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40196 (local address 127.0.97.60:37393)
0928 08:44:37.053965 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:37.053970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:37.054531 (+ 561us) server_negotiation.cc:407] Connection header received
0928 08:44:37.054620 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:37.054625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:37.054753 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:37.055842 (+ 1089us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:37.056037 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.057498 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.059269 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.060013 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.060727 (+ 714us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:37.064067 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:37.064073 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:37.064078 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:37.064118 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:37.067514 (+ 3396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:37.068693 (+ 1179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:37.068699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:37.068703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:37.068826 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:37.069307 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:37.069314 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:37.069319 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:37.069607 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:37.069741 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:37.070172 (+ 431us) server_negotiation.cc:299] Negotiation successful
0928 08:44:37.070383 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 08:44:37.070763 7273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:37.053640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37393 (local address 127.0.0.1:40196)
0928 08:44:37.054298 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:37.054317 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:37.054425 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:37.054922 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:37.054927 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:37.055412 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:37.055616 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:37.055628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.057657 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.057664 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:37.059085 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:37.059100 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.060191 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.060199 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:37.060383 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:37.061139 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:37.061161 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:37.063858 (+ 2697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:37.067652 (+ 3794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:37.067658 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:37.067662 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:37.068563 (+ 901us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:37.069034 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:37.069043 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:37.069046 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:37.069192 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:37.069706 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:37.069711 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:37.069869 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 08:44:37.070050 (+ 181us) client_negotiation.cc:240] Negotiation successful
0928 08:44:37.070346 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260928 08:44:37.072299 6552 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.97.60:37393: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a63285bae4b64ecca10248bb821d965f"
member_type: NON_VOTER
last_known_addr {
host: "127.0.97.60"
port: 37393
}
attrs {
promote: true
}
}
dest_uuid: "0c813b2df2b645b18971acadaa4ca63a"
cas_config_opid_index: 5
I20260928 08:44:37.073421 6552 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: true } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37393 } attrs { promote: true } }
Sep 28 08:44:37 dist-test-slave-n6xs krb5kdc[6507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260928 08:44:37.076045 7272 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a63285bae4b64ecca10248bb821d965f (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: true } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37393 } attrs { promote: true } } }
I20260928 08:44:37.076570 7160 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [term 1 LEARNER]: Refusing update from remote peer 0c813b2df2b645b18971acadaa4ca63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 08:44:37.083163 7241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:37.086792 7299 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 08:44:37.088238 7241 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [term 1 FOLLOWER]: Refusing update from remote peer 0c813b2df2b645b18971acadaa4ca63a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 08:44:37.090432 7300 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37393 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:44:37.091230 7272 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.97.60:37393
I20260928 08:44:37.091964 7114 master.cc:537] Master@127.0.97.60:37393 shutting down...
I20260928 08:44:37.098918 7300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0c813b2df2b645b18971acadaa4ca63a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: true } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37393 } attrs { promote: true } } }
I20260928 08:44:37.099220 7300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:37.101612 7160 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a63285bae4b64ecca10248bb821d965f (127.0.97.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: true } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37393 } attrs { promote: true } } }
I20260928 08:44:37.105907 7188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0c813b2df2b645b18971acadaa4ca63a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: NON_VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: true } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37393 } attrs { promote: true } } }
I20260928 08:44:37.106163 7188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [sys.catalog]: This master's current role is: LEARNER
I20260928 08:44:37.107475 7297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:37.050277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57560 (local address 127.0.97.1:46847)
0928 08:44:37.062997 (+ 12720us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:37.063007 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:37.063040 (+ 33us) server_negotiation.cc:407] Connection header received
0928 08:44:37.063107 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:37.063118 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:37.063419 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:37.063611 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:37.065311 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.066961 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.069027 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.069997 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.071362 (+ 1365us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:37.091281 (+ 19919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:37.091361 (+ 80us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:37.091383 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:37.091434 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:37.097163 (+ 5729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:37.104506 (+ 7343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:37.104538 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:37.104545 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:37.104740 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:37.105302 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:37.105307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:37.105311 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:37.105726 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:37.106047 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:37.106738 (+ 691us) server_negotiation.cc:299] Negotiation successful
0928 08:44:37.107052 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12499,"thread_start_us":13552,"threads_started":1}
I20260928 08:44:37.108937 6736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:37.112350 7272 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a: attempting to promote NON_VOTER 17003f6ce92c4415a110af3b597c8e4a to VOTER
I20260928 08:44:37.113447 7272 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: false } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37393 } attrs { promote: true } }
I20260928 08:44:37.117974 7160 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [term 1 LEARNER]: Refusing update from remote peer 0c813b2df2b645b18971acadaa4ca63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 08:44:37.118669 7300 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 08:44:37.121356 7300 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 17003f6ce92c4415a110af3b597c8e4a (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: false } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37393 } attrs { promote: true } } }
I20260928 08:44:37.122030 7160 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 17003f6ce92c4415a110af3b597c8e4a (127.0.97.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: false } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37393 } attrs { promote: true } } }
I20260928 08:44:37.126281 7298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0c813b2df2b645b18971acadaa4ca63a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: false } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37393 } attrs { promote: true } } }
I20260928 08:44:37.126655 7298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:37.126873 7272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0c813b2df2b645b18971acadaa4ca63a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: false } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37393 } attrs { promote: true } } }
I20260928 08:44:37.126993 7277 raft_consensus.cc:482] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:44:37.127072 7272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:37.127243 7277 raft_consensus.cc:504] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37393 } }
W20260928 08:44:37.133756 6539 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a -> Peer a63285bae4b64ecca10248bb821d965f (127.0.97.60:37393): Couldn't send request to peer a63285bae4b64ecca10248bb821d965f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 08:44:37.135926 7188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0c813b2df2b645b18971acadaa4ca63a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: false } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37393 } attrs { promote: true } } }
I20260928 08:44:37.136183 7188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:44:37.141193 7277 leader_election.cc:288] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c813b2df2b645b18971acadaa4ca63a (127.0.97.62:46083), 17003f6ce92c4415a110af3b597c8e4a (127.0.97.61:36547)
W20260928 08:44:37.141599 7220 tablet.cc:2369] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:44:37.142297 7160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a63285bae4b64ecca10248bb821d965f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17003f6ce92c4415a110af3b597c8e4a" is_pre_election: true
I20260928 08:44:37.142848 6562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a63285bae4b64ecca10248bb821d965f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c813b2df2b645b18971acadaa4ca63a" is_pre_election: true
I20260928 08:44:37.143747 7216 leader_election.cc:302] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a63285bae4b64ecca10248bb821d965f; no voters: 0c813b2df2b645b18971acadaa4ca63a, 17003f6ce92c4415a110af3b597c8e4a
I20260928 08:44:37.144652 7277 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:44:37.145730 7114 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:44:37.146100 7114 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:37.146284 7114 tablet_replica.cc:330] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f: stopping tablet replica
I20260928 08:44:37.160101 7114 master.cc:552] Master@127.0.97.60:37393 shutdown complete.
I20260928 08:44:37.160193 7114 master_runner.cc:305] Clearing existing system tablet
I20260928 08:44:37.160601 7114 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:37.160717 7114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:37.164093 7114 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260928 08:44:37.165736 7306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:37.166062 7114 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:37.166468 7114 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 08:44:37.166606 7114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-2/wal
uuid: "a63285bae4b64ecca10248bb821d965f"
format_stamp: "Formatted at 2026-09-28 08:44:36 on dist-test-slave-n6xs"
server_key: "0282e586ebcd48df2158fffc111f0ba6"
server_key_iv: "0bbca4511b46d5b4f8e0657f40c1d882"
server_key_version: "encryptionkey@0"
I20260928 08:44:37.172297 7114 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:37.179428 7114 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 08:44:37.179699 7114 log.cc:1196] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 08:44:37.180354 7114 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f: Deleting consensus metadata
I20260928 08:44:37.180657 7114 master_runner.cc:315] Copying system tablet from 127.0.97.62:46083
I20260928 08:44:37.183169 7114 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.97.62:46083
I20260928 08:44:37.204649 7271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:37.184094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35970 (local address 127.0.97.62:46083)
0928 08:44:37.184316 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:37.184324 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:37.189132 (+ 4808us) server_negotiation.cc:407] Connection header received
0928 08:44:37.189419 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:37.189425 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:37.189516 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:37.189584 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:37.190843 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.192199 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.194073 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.194720 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.194813 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:37.197365 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:37.197372 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:37.197389 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:37.197426 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:37.201428 (+ 4002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:37.202285 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:37.202291 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:37.202294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:37.202443 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:37.202996 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:37.203001 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:37.203005 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:37.203368 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:37.203513 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:37.204076 (+ 563us) server_negotiation.cc:299] Negotiation successful
0928 08:44:37.204286 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 08:44:37.205966 7312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:37.184002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.0.1:35970)
0928 08:44:37.189001 (+ 4999us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:37.189027 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:37.189255 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:37.189814 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:37.189819 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:37.190352 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:37.190620 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:37.190631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.192389 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.192395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:37.193915 (+ 1520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:37.193930 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.195094 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.195100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:37.195307 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:37.196067 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:37.196093 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:37.197160 (+ 1067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:37.201639 (+ 4479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:37.201653 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:37.201659 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:37.202138 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:37.202589 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:37.202593 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:37.202596 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:37.202866 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:37.203603 (+ 737us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:37.203608 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:37.203770 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 08:44:37.205180 (+ 1410us) client_negotiation.cc:240] Negotiation successful
0928 08:44:37.205522 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4855,"thread_start_us":4705,"threads_started":1}
I20260928 08:44:37.206792 6572 tablet_copy_service.cc:140] P 0c813b2df2b645b18971acadaa4ca63a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a63285bae4b64ecca10248bb821d965f ({username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:35970)
I20260928 08:44:37.206948 6572 tablet_copy_service.cc:161] P 0c813b2df2b645b18971acadaa4ca63a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a63285bae4b64ecca10248bb821d965f at {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:35970: session id = a63285bae4b64ecca10248bb821d965f-00000000000000000000000000000000
I20260928 08:44:37.210525 6572 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 08:44:37.212401 7114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 08:44:37.220000 7114 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f: tablet copy: Starting download of 0 data blocks...
I20260928 08:44:37.220306 7114 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f: tablet copy: Starting download of 1 WAL segments...
I20260928 08:44:37.222900 7114 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 08:44:37.227226 6572 tablet_copy_service.cc:342] P 0c813b2df2b645b18971acadaa4ca63a: Request end of tablet copy session a63285bae4b64ecca10248bb821d965f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:35970
I20260928 08:44:37.227421 6572 tablet_copy_service.cc:433] P 0c813b2df2b645b18971acadaa4ca63a: ending tablet copy session a63285bae4b64ecca10248bb821d965f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a63285bae4b64ecca10248bb821d965f
I20260928 08:44:37.237975 7114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:37.240535 7313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:37.247223 7314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:37.251451 7316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:37.252530 7114 server_base.cc:1034] running on GCE node
I20260928 08:44:37.252835 7114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:37.253479 7114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:37.254714 7114 hybrid_clock.cc:648] HybridClock initialized: now 1790585077254662 us; error 39 us; skew 500 ppm
I20260928 08:44:37.254969 7114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:37 dist-test-slave-n6xs krb5kdc[6507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585077, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:37.257102 7114 init.cc:377] Logged in from keytab as oryx/127.0.97.60@KRBTEST.COM (short username oryx)
I20260928 08:44:37.262929 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 6613
I20260928 08:44:37.268317 7114 webserver.cc:460] Webserver started at http://127.0.97.60:46819/ using document root <none> and password file <none>
I20260928 08:44:37.268641 7114 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:37.268754 7114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:37.271538 7114 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260928 08:44:37.273247 7323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:37.273670 7114 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 08:44:37.273792 7114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-2/wal
uuid: "a63285bae4b64ecca10248bb821d965f"
format_stamp: "Formatted at 2026-09-28 08:44:36 on dist-test-slave-n6xs"
server_key: "0282e586ebcd48df2158fffc111f0ba6"
server_key_iv: "0bbca4511b46d5b4f8e0657f40c1d882"
server_key_version: "encryptionkey@0"
I20260928 08:44:37.273980 7114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:37.287253 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:46847
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46083
--webserver_port=41501
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:46083,127.0.97.61:36547
--tserver_master_addrs=127.0.97.62:46083,127.0.97.61:36547,127.0.97.60:37393 with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:44:37.295207 7114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:37.298717 7114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:37.332949 7114 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:37393
I20260928 08:44:37.332980 7375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:37393 every 8 connection(s)
I20260928 08:44:37.333910 7114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/master-2/data/info.pb
I20260928 08:44:37.342144 7376 sys_catalog.cc:263] Verifying existing consensus state
I20260928 08:44:37.348271 7376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f: Bootstrap starting.
I20260928 08:44:37.363740 7376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:44:37.364248 7376 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f: Bootstrap complete.
I20260928 08:44:37.365099 7376 raft_consensus.cc:348] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: false } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37393 } attrs { promote: true } }
I20260928 08:44:37.365342 7376 raft_consensus.cc:729] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a63285bae4b64ecca10248bb821d965f, State: Initialized, Role: LEARNER
I20260928 08:44:37.365594 7376 consensus_queue.cc:260] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: false } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37393 } attrs { promote: true } }
I20260928 08:44:37.366331 7379 sys_catalog.cc:455] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: false } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37393 } attrs { promote: true } } }
I20260928 08:44:37.366652 7376 sys_catalog.cc:564] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:44:37.366854 7379 sys_catalog.cc:458] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [sys.catalog]: This master's current role is: LEARNER
I20260928 08:44:37.377007 7390 catalog_manager.cc:1259] Loaded cluster ID: cd79b6032ebe422ab25f6fc91d390632
I20260928 08:44:37.377133 7390 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f: loading cluster ID for follower catalog manager: success
I20260928 08:44:37.379398 7390 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f: acquiring CA information for follower catalog manager: success
I20260928 08:44:37.381213 7390 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 08:44:37.653839 7392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:37.646476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40198 (local address 127.0.97.60:37393)
0928 08:44:37.647128 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:37.647134 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:37.647359 (+ 225us) server_negotiation.cc:407] Connection header received
0928 08:44:37.647559 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:37.647563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:37.647631 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:37.647733 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:37.648231 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.649162 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.651393 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.652561 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.652661 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:37.652846 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:37.653446 (+ 600us) server_negotiation.cc:299] Negotiation successful
0928 08:44:37.653561 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":349,"threads_started":1}
I20260928 08:44:37.653923 7391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:37.646321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37393 (local address 127.0.0.1:40198)
0928 08:44:37.647237 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:37.647261 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:37.647429 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:37.647782 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:37.647787 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:37.647835 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:37.648072 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:37.648085 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.649305 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.649311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:37.651231 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:37.651249 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.653035 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.653042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:37.653236 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:37.653282 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 08:44:37.653396 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 08:44:37.653556 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"thread_start_us":705,"threads_started":1}
I20260928 08:44:37.654861 7351 raft_consensus.cc:1203] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 08:44:37.655463 7379 sys_catalog.cc:455] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c813b2df2b645b18971acadaa4ca63a. Latest consensus state: current_term: 1 leader_uuid: "0c813b2df2b645b18971acadaa4ca63a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: false } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37393 } attrs { promote: true } } }
I20260928 08:44:37.655795 7379 sys_catalog.cc:458] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [sys.catalog]: This master's current role is: LEARNER
W20260928 08:44:37.717937 7328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:37.718246 7328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:37.718416 7328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:37.726049 7328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:37.726200 7328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:37.726387 7328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:44:37.736445 7328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:46847
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=41501
--tserver_master_addrs=127.0.97.62:46083,127.0.97.61:36547,127.0.97.60:37393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:37.737792 7328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:37.739112 7328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:37.753000 7398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:37.752939 7397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:37.753240 7400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:37.754170 7328 server_base.cc:1034] running on GCE node
I20260928 08:44:37.754798 7328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:37.755875 7328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:37.757093 7328 hybrid_clock.cc:648] HybridClock initialized: now 1790585077757055 us; error 38 us; skew 500 ppm
I20260928 08:44:37.757437 7328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:37 dist-test-slave-n6xs krb5kdc[6507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585077, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:37.760680 7328 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260928 08:44:37.762243 7328 webserver.cc:460] Webserver started at http://127.0.97.1:41501/ using document root <none> and password file <none>
I20260928 08:44:37.762951 7328 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:37.763108 7328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:37.769104 7328 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 08:44:37.772017 7407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:37.773401 7328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:44:37.773604 7328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-0/wal
uuid: "fb125058c6c34aae8dff092e2f334d3b"
format_stamp: "Formatted at 2026-09-28 08:44:34 on dist-test-slave-n6xs"
server_key: "6f17a249f864445954cc0ebb8c947195"
server_key_iv: "c039a0acfef6d4227718eefdf66ebaca"
server_key_version: "encryptionkey@0"
I20260928 08:44:37.774662 7328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:37.801666 7328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:37.806191 7328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:37.806509 7328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:37.807538 7328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:44:37.809399 7328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:44:37.809523 7328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:37.809660 7328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:44:37.809748 7328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:37.843416 7328 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:46847
I20260928 08:44:37.843511 7519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:46847 every 8 connection(s)
I20260928 08:44:37.845098 7328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:44:37.845701 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 7328
I20260928 08:44:37.846024 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 6747
Sep 28 08:44:37 dist-test-slave-n6xs krb5kdc[6507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260928 08:44:37.888902 7524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:37.847296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53072 (local address 127.0.97.2:42561)
0928 08:44:37.858668 (+ 11372us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:37.858680 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:37.859841 (+ 1161us) server_negotiation.cc:407] Connection header received
0928 08:44:37.859940 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:37.859953 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:37.860298 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:37.860480 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:37.863684 (+ 3204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.865753 (+ 2069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.868301 (+ 2548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.869704 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.869873 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:37.878557 (+ 8684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:37.878578 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:37.878600 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:37.878657 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:37.883416 (+ 4759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:37.884521 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:37.884536 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:37.884543 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:37.884748 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:37.885805 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:37.885811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:37.885814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:37.886262 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:37.886582 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:37.887912 (+ 1330us) server_negotiation.cc:299] Negotiation successful
0928 08:44:37.888270 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11164,"thread_start_us":11003,"threads_started":1}
I20260928 08:44:37.890933 6870 generic_service.cc:196] Checking for leaks (request via RPC)
Sep 28 08:44:37 dist-test-slave-n6xs krb5kdc[6507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585077, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 28 08:44:37 dist-test-slave-n6xs krb5kdc[6507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585077, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 28 08:44:37 dist-test-slave-n6xs krb5kdc[6507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585077, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260928 08:44:37.906873 7525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:37.848323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39069 (local address 127.0.97.62:46083)
0928 08:44:37.863847 (+ 15524us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:37.863857 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:37.878598 (+ 14741us) server_negotiation.cc:407] Connection header received
0928 08:44:37.882807 (+ 4209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:37.882814 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:37.882913 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:37.883058 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:37.886893 (+ 3835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.888090 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.890684 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.891348 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.891447 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:37.898075 (+ 6628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:37.898085 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:37.898093 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:37.898149 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:37.902739 (+ 4590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:37.903986 (+ 1247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:37.903994 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:37.904000 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:37.904181 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:37.904691 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:37.904698 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:37.904704 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:37.905134 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:37.905295 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:37.905983 (+ 688us) server_negotiation.cc:299] Negotiation successful
0928 08:44:37.906289 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15332,"thread_start_us":15536,"threads_started":1}
I20260928 08:44:37.907035 7529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:37.867108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41681 (local address 127.0.97.61:36547)
0928 08:44:37.882420 (+ 15312us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:37.882429 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:37.883582 (+ 1153us) server_negotiation.cc:407] Connection header received
0928 08:44:37.883655 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:37.883660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:37.883757 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:37.883892 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:37.888237 (+ 4345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.889318 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.893109 (+ 3791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.893762 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.893881 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:37.898463 (+ 4582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:37.898470 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:37.898475 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:37.898515 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:37.902213 (+ 3698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:37.903563 (+ 1350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:37.903568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:37.903573 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:37.903722 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:37.904315 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:37.904321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:37.904327 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:37.904709 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:37.904854 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:37.906427 (+ 1573us) server_negotiation.cc:299] Negotiation successful
0928 08:44:37.906642 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15176,"thread_start_us":15328,"threads_started":1}
I20260928 08:44:37.908499 7530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:37.867057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36547 (local address 127.0.97.1:41681)
0928 08:44:37.883312 (+ 16255us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:37.883331 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:37.883447 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:37.884511 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:37.884515 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:37.885121 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:37.886376 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:37.886406 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.890506 (+ 4100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.890521 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:37.892937 (+ 2416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:37.892952 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.894150 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.894158 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:37.894375 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:37.895063 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:37.895084 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:37.898215 (+ 3131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:37.902576 (+ 4361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:37.902589 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:37.902609 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:37.903417 (+ 808us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:37.903906 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:37.903911 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:37.903915 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:37.904157 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:37.904928 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:37.904939 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:37.905408 (+ 469us) client_negotiation.cc:769] Sending connection context
0928 08:44:37.905808 (+ 400us) client_negotiation.cc:240] Negotiation successful
0928 08:44:37.906848 (+ 1040us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16101,"mutex_wait_us":829,"spinlock_wait_cycles":37632,"thread_start_us":15965,"threads_started":1}
I20260928 08:44:37.908542 7526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:37.848719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.97.1:39069)
0928 08:44:37.878439 (+ 29720us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:37.878474 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:37.882637 (+ 4163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:37.883541 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:37.883554 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:37.884928 (+ 1374us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:37.886733 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:37.886742 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.888278 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.888296 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:37.890506 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:37.890517 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.891786 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.891792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:37.892156 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:37.893182 (+ 1026us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:37.893225 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:37.897702 (+ 4477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:37.902971 (+ 5269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:37.902979 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:37.902986 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:37.903745 (+ 759us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:37.904354 (+ 609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:37.904362 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:37.904368 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:37.904548 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:37.905389 (+ 841us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:37.905397 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:37.905601 (+ 204us) client_negotiation.cc:769] Sending connection context
0928 08:44:37.907084 (+ 1483us) client_negotiation.cc:240] Negotiation successful
0928 08:44:37.907383 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29521,"mutex_wait_us":1631,"thread_start_us":27586,"threads_started":1}
I20260928 08:44:37.910157 7521 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46083
I20260928 08:44:37.910483 7521 heartbeater.cc:464] Registering TS with master...
I20260928 08:44:37.910854 7392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:37.848953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:55289 (local address 127.0.97.60:37393)
0928 08:44:37.849180 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:37.849189 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:37.876653 (+ 27464us) server_negotiation.cc:407] Connection header received
0928 08:44:37.886962 (+ 10309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:37.886970 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:37.887063 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:37.887133 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:37.892667 (+ 5534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.893712 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.895850 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.897022 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.897132 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:37.902535 (+ 5403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:37.902548 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:37.902554 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:37.902601 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:37.905896 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:37.907573 (+ 1677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:37.907581 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:37.907588 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:37.907733 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:37.908347 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:37.908351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:37.908354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:37.908685 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:37.908819 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:37.910278 (+ 1459us) server_negotiation.cc:299] Negotiation successful
0928 08:44:37.910525 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 08:44:37.911393 7521 heartbeater.cc:511] Master 127.0.97.62:46083 requested a full tablet report, sending...
I20260928 08:44:37.911962 7522 heartbeater.cc:347] Connected to a master server at 127.0.97.61:36547
I20260928 08:44:37.912436 7522 heartbeater.cc:464] Registering TS with master...
I20260928 08:44:37.913033 7522 heartbeater.cc:511] Master 127.0.97.61:36547 requested a full tablet report, sending...
I20260928 08:44:37.912431 7527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:37.848719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37393 (local address 127.0.97.1:55289)
0928 08:44:37.876396 (+ 27677us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:37.876484 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:37.882203 (+ 5719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:37.887328 (+ 5125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:37.887332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:37.887770 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:37.888045 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:37.888056 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.893913 (+ 5857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.893918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:37.895627 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:37.895643 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:37.897428 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:37.897436 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:37.897640 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:37.898255 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:37.898280 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:37.901705 (+ 3425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:37.906171 (+ 4466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:37.906175 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:37.906178 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:37.906594 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:37.907963 (+ 1369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:37.907967 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:37.907971 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:37.908147 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:37.909466 (+ 1319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:37.909470 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:37.909793 (+ 323us) client_negotiation.cc:769] Sending connection context
0928 08:44:37.911217 (+ 1424us) client_negotiation.cc:240] Negotiation successful
0928 08:44:37.911506 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27124,"thread_start_us":35224,"threads_started":1}
I20260928 08:44:37.913904 6552 ts_manager.cc:194] Re-registered known tserver with Master: fb125058c6c34aae8dff092e2f334d3b (127.0.97.1:46847)
I20260928 08:44:37.914371 7520 heartbeater.cc:347] Connected to a master server at 127.0.97.60:37393
I20260928 08:44:37.914526 7520 heartbeater.cc:464] Registering TS with master...
I20260928 08:44:37.914848 7520 heartbeater.cc:511] Master 127.0.97.60:37393 requested a full tablet report, sending...
I20260928 08:44:37.915539 7150 ts_manager.cc:194] Registered new tserver with Master: fb125058c6c34aae8dff092e2f334d3b (127.0.97.1:46847)
I20260928 08:44:37.915952 6552 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39069
I20260928 08:44:37.916476 7341 ts_manager.cc:194] Registered new tserver with Master: fb125058c6c34aae8dff092e2f334d3b (127.0.97.1:46847)
I20260928 08:44:37.996976 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 6747
I20260928 08:44:38.015683 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:42561
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46083
--webserver_port=34735
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:46083,127.0.97.61:36547
--tserver_master_addrs=127.0.97.62:46083,127.0.97.61:36547,127.0.97.60:37393 with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:44:38.249699 7272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0c813b2df2b645b18971acadaa4ca63a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: false } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: NON_VOTER last_known_addr { host: "127.0.97.60" port: 37393 } attrs { promote: true } } }
I20260928 08:44:38.250003 7272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:38.251221 7533 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a: attempting to promote NON_VOTER a63285bae4b64ecca10248bb821d965f to VOTER
I20260928 08:44:38.252007 7533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: false } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37393 } attrs { promote: false } }
I20260928 08:44:38.254240 7351 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [term 1 LEARNER]: Refusing update from remote peer 0c813b2df2b645b18971acadaa4ca63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 08:44:38.254413 7160 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [term 1 FOLLOWER]: Refusing update from remote peer 0c813b2df2b645b18971acadaa4ca63a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 08:44:38.254812 7272 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 08:44:38.255223 7534 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [LEADER]: Connected to new peer: Peer: permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 08:44:38.259521 7272 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a63285bae4b64ecca10248bb821d965f (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: false } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37393 } attrs { promote: false } } }
I20260928 08:44:38.260360 7351 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a63285bae4b64ecca10248bb821d965f (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: false } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37393 } attrs { promote: false } } }
I20260928 08:44:38.263180 7272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0c813b2df2b645b18971acadaa4ca63a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: false } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37393 } attrs { promote: false } } }
I20260928 08:44:38.263414 7272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:38.263649 7533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0c813b2df2b645b18971acadaa4ca63a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: false } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37393 } attrs { promote: false } } }
I20260928 08:44:38.264345 7533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:38.264358 7160 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a63285bae4b64ecca10248bb821d965f (127.0.97.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: false } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37393 } attrs { promote: false } } }
I20260928 08:44:38.268853 7534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0c813b2df2b645b18971acadaa4ca63a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: false } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37393 } attrs { promote: false } } }
I20260928 08:44:38.269258 7534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:38.269971 7540 sys_catalog.cc:455] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0c813b2df2b645b18971acadaa4ca63a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: false } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37393 } attrs { promote: false } } }
I20260928 08:44:38.270496 7540 sys_catalog.cc:458] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:44:38.271054 7541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0c813b2df2b645b18971acadaa4ca63a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: false } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37393 } attrs { promote: false } } }
I20260928 08:44:38.271483 7541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [sys.catalog]: This master's current role is: FOLLOWER
W20260928 08:44:38.354064 7532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:38.354367 7532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:38.354512 7532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:38.361797 7532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:38.361917 7532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:38.362028 7532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:44:38.372156 7532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:42561
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=34735
--tserver_master_addrs=127.0.97.62:46083,127.0.97.61:36547,127.0.97.60:37393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:38.373420 7532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:38.374711 7532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:38.388206 7547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:38.388242 7549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:38.388346 7546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:38.388973 7532 server_base.cc:1034] running on GCE node
I20260928 08:44:38.389528 7532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:38.390489 7532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:38.391717 7532 hybrid_clock.cc:648] HybridClock initialized: now 1790585078391659 us; error 49 us; skew 500 ppm
I20260928 08:44:38.392068 7532 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:38 dist-test-slave-n6xs krb5kdc[6507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585078, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:38.395260 7532 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260928 08:44:38.396726 7532 webserver.cc:460] Webserver started at http://127.0.97.2:34735/ using document root <none> and password file <none>
I20260928 08:44:38.397330 7532 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:38.397433 7532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:38.402386 7532 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 08:44:38.404950 7556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:38.406294 7532 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 08:44:38.406455 7532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-1/wal
uuid: "1137082ae436457c84ba8100f189c4ec"
format_stamp: "Formatted at 2026-09-28 08:44:35 on dist-test-slave-n6xs"
server_key: "0cfe32ca52337c930027c9615ea2c7a3"
server_key_iv: "eff70e0cdcad695bb88cc7c40d30ed21"
server_key_version: "encryptionkey@0"
I20260928 08:44:38.407502 7532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:38.428574 7532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:38.432358 7532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:38.432619 7532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:38.433529 7532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:44:38.434885 7532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:44:38.434971 7532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:38.435134 7532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:44:38.435233 7532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:38.467887 7532 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:42561
I20260928 08:44:38.467991 7668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:42561 every 8 connection(s)
I20260928 08:44:38.469408 7532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:44:38.476831 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 7532
I20260928 08:44:38.477139 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 6880
Sep 28 08:44:38 dist-test-slave-n6xs krb5kdc[6507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585078, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 28 08:44:38 dist-test-slave-n6xs krb5kdc[6507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585078, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
Sep 28 08:44:38 dist-test-slave-n6xs krb5kdc[6507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260928 08:44:38.511711 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:38.473648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44609 (local address 127.0.97.61:36547)
0928 08:44:38.488595 (+ 14947us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:38.488599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:38.488615 (+ 16us) server_negotiation.cc:407] Connection header received
0928 08:44:38.488662 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:38.488666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:38.488720 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:38.488844 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:38.491637 (+ 2793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:38.492735 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:38.495206 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:38.495844 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:38.495965 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:38.503188 (+ 7223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:38.503195 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:38.503200 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:38.503252 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:38.506876 (+ 3624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:38.507885 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:38.507891 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:38.507895 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:38.508042 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:38.509401 (+ 1359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:38.509405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:38.509408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:38.509777 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:38.509925 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:38.511066 (+ 1141us) server_negotiation.cc:299] Negotiation successful
0928 08:44:38.511314 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14857,"thread_start_us":14702,"threads_started":1}
I20260928 08:44:38.514148 7675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:38.472981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36547 (local address 127.0.97.2:44609)
0928 08:44:38.484473 (+ 11492us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:38.484522 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:38.487671 (+ 3149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:38.489144 (+ 1473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:38.489160 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:38.490482 (+ 1322us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:38.491430 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:38.491456 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:38.492956 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:38.492962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:38.495040 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:38.495058 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:38.496344 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:38.496350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:38.496706 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:38.497582 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:38.497611 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:38.501346 (+ 3735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:38.507071 (+ 5725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:38.507080 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:38.507099 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:38.507719 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:38.508189 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:38.508193 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:38.508197 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:38.508438 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:38.510031 (+ 1593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:38.510041 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:38.510577 (+ 536us) client_negotiation.cc:769] Sending connection context
0928 08:44:38.512143 (+ 1566us) client_negotiation.cc:240] Negotiation successful
0928 08:44:38.512691 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11173,"mutex_wait_us":36,"thread_start_us":10895,"threads_started":1}
I20260928 08:44:38.516207 7671 heartbeater.cc:347] Connected to a master server at 127.0.97.61:36547
I20260928 08:44:38.516559 7671 heartbeater.cc:464] Registering TS with master...
Sep 28 08:44:38 dist-test-slave-n6xs krb5kdc[6507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585078, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 08:44:38.517405 7674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:38.472300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:53269 (local address 127.0.97.60:37393)
0928 08:44:38.483848 (+ 11548us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:38.483856 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:38.491295 (+ 7439us) server_negotiation.cc:407] Connection header received
0928 08:44:38.491429 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:38.491437 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:38.491536 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:38.491637 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:38.493082 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:38.494397 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:38.496916 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:38.497786 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:38.497903 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:38.509461 (+ 11558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:38.509471 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:38.509478 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:38.509532 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:38.513808 (+ 4276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:38.514620 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:38.514626 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:38.514630 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:38.514764 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:38.515584 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:38.515590 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:38.515594 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:38.515979 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:38.516151 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:38.516829 (+ 678us) server_negotiation.cc:299] Negotiation successful
0928 08:44:38.517029 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11372,"thread_start_us":15797,"threads_started":1}
I20260928 08:44:38.517498 7671 heartbeater.cc:511] Master 127.0.97.61:36547 requested a full tablet report, sending...
I20260928 08:44:38.518358 7676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:38.472982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37393 (local address 127.0.97.2:53269)
0928 08:44:38.491160 (+ 18178us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:38.491179 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:38.491304 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:38.491853 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:38.491860 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:38.492546 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:38.492880 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:38.492895 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:38.494605 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:38.494613 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:38.496698 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:38.496717 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:38.498230 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:38.498239 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:38.500137 (+ 1898us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:38.501756 (+ 1619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:38.501778 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:38.509232 (+ 7454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:38.513950 (+ 4718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:38.513954 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:38.513958 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:38.514468 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:38.514939 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:38.514944 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:38.514948 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:38.515415 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:38.516250 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:38.516273 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:38.516472 (+ 199us) client_negotiation.cc:769] Sending connection context
0928 08:44:38.517632 (+ 1160us) client_negotiation.cc:240] Negotiation successful
0928 08:44:38.517910 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18042,"mutex_wait_us":498,"thread_start_us":16784,"threads_started":1}
I20260928 08:44:38.519151 7669 heartbeater.cc:347] Connected to a master server at 127.0.97.60:37393
I20260928 08:44:38.519318 7669 heartbeater.cc:464] Registering TS with master...
I20260928 08:44:38.519594 7150 ts_manager.cc:194] Registered new tserver with Master: 1137082ae436457c84ba8100f189c4ec (127.0.97.2:42561)
I20260928 08:44:38.519740 7669 heartbeater.cc:511] Master 127.0.97.60:37393 requested a full tablet report, sending...
I20260928 08:44:38.520859 7341 ts_manager.cc:194] Registered new tserver with Master: 1137082ae436457c84ba8100f189c4ec (127.0.97.2:42561)
I20260928 08:44:38.527190 7673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:38.472255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40319 (local address 127.0.97.62:46083)
0928 08:44:38.496340 (+ 24085us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:38.496346 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:38.496362 (+ 16us) server_negotiation.cc:407] Connection header received
0928 08:44:38.496425 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:38.496430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:38.496508 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:38.497744 (+ 1236us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:38.497885 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:38.499285 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:38.505662 (+ 6377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:38.506337 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:38.518991 (+ 12654us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:38.520797 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:38.520803 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:38.520808 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:38.520863 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:38.524504 (+ 3641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:38.525213 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:38.525218 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:38.525223 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:38.525365 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:38.525755 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:38.525759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:38.525761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:38.526072 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:38.526215 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:38.526608 (+ 393us) server_negotiation.cc:299] Negotiation successful
0928 08:44:38.526843 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23967,"thread_start_us":23839,"threads_started":1}
I20260928 08:44:38.527195 7677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:38.472981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.97.2:40319)
0928 08:44:38.490563 (+ 17582us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:38.490581 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:38.490698 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:38.496676 (+ 5978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:38.496680 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:38.497243 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:38.497595 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:38.497607 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:38.503041 (+ 5434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:38.503050 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:38.504677 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:38.504692 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:38.514191 (+ 9499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:38.514198 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:38.514662 (+ 464us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:38.515250 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:38.515272 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:38.518365 (+ 3093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:38.524697 (+ 6332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:38.524701 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:38.524704 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:38.525052 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:38.525487 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:38.525492 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:38.525496 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:38.525625 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:38.526194 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:38.526199 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:38.526371 (+ 172us) client_negotiation.cc:769] Sending connection context
0928 08:44:38.526499 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 08:44:38.526789 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17434,"mutex_wait_us":37,"spinlock_wait_cycles":19200,"thread_start_us":17693,"threads_started":1}
I20260928 08:44:38.528002 7670 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46083
I20260928 08:44:38.528149 7670 heartbeater.cc:464] Registering TS with master...
I20260928 08:44:38.528220 7680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:38.478076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36810 (local address 127.0.97.3:41011)
0928 08:44:38.491878 (+ 13802us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:38.491886 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:38.491912 (+ 26us) server_negotiation.cc:407] Connection header received
0928 08:44:38.491978 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:38.491987 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:38.492211 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:38.494184 (+ 1973us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:38.494396 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:38.495913 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:38.501955 (+ 6042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:38.502868 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:38.513014 (+ 10146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:38.513271 (+ 257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:38.513282 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:38.513297 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:38.513325 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:38.524939 (+ 11614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:38.525636 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:38.525645 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:38.525649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:38.525781 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:38.526169 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:38.526176 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:38.526179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:38.526505 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:38.526787 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:38.527101 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 08:44:38.527732 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13593,"thread_start_us":13438,"threads_started":1}
I20260928 08:44:38.528581 7670 heartbeater.cc:511] Master 127.0.97.62:46083 requested a full tablet report, sending...
I20260928 08:44:38.529538 6552 ts_manager.cc:194] Re-registered known tserver with Master: 1137082ae436457c84ba8100f189c4ec (127.0.97.2:42561)
I20260928 08:44:38.529623 7003 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:38.530362 6552 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:40319
I20260928 08:44:38.613304 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 6880
I20260928 08:44:38.629319 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:41011
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46083
--webserver_port=46489
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:43867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.97.62:46083,127.0.97.61:36547
--tserver_master_addrs=127.0.97.62:46083,127.0.97.61:36547,127.0.97.60:37393 with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:44:38.918911 7521 heartbeater.cc:503] Master 127.0.97.62:46083 was elected leader, sending a full tablet report...
W20260928 08:44:38.949856 7682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:38.950126 7682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:38.950290 7682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:38.957535 7682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:38.957659 7682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:38.957829 7682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:44:38.967794 7682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:41011
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=46489
--tserver_master_addrs=127.0.97.62:46083,127.0.97.61:36547,127.0.97.60:37393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:38.969251 7682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:38.970708 7682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:38.983778 7687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:38.983855 7688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:38.983906 7690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:38.984591 7682 server_base.cc:1034] running on GCE node
I20260928 08:44:38.985200 7682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:38.986099 7682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:38.987326 7682 hybrid_clock.cc:648] HybridClock initialized: now 1790585078987270 us; error 71 us; skew 500 ppm
I20260928 08:44:38.987634 7682 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:38 dist-test-slave-n6xs krb5kdc[6507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585078, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:38.990659 7682 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260928 08:44:38.992123 7682 webserver.cc:460] Webserver started at http://127.0.97.3:46489/ using document root <none> and password file <none>
I20260928 08:44:38.992722 7682 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:38.992851 7682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:38.997905 7682 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260928 08:44:39.000520 7697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:39.001907 7682 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:44:39.002053 7682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-2/wal
uuid: "be8863ce64e24035aaeed4bf282343b3"
format_stamp: "Formatted at 2026-09-28 08:44:35 on dist-test-slave-n6xs"
server_key: "d7fd1951a7c8aebeb5be9678e5e58f80"
server_key_iv: "c5b9e6257c2822b706894c7a92215875"
server_key_version: "encryptionkey@0"
I20260928 08:44:39.002979 7682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:39.021370 7682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:39.025549 7682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:39.025835 7682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:39.026757 7682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:44:39.028263 7682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:44:39.028383 7682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:39.028506 7682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:44:39.028590 7682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:39.063091 7682 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:41011
I20260928 08:44:39.063177 7809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:41011 every 8 connection(s)
I20260928 08:44:39.064558 7682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:44:39.068642 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 7682
Sep 28 08:44:39 dist-test-slave-n6xs krb5kdc[6507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585078, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
Sep 28 08:44:39 dist-test-slave-n6xs krb5kdc[6507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585078, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.61@KRBTEST.COM
Sep 28 08:44:39 dist-test-slave-n6xs krb5kdc[6507](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585078, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.60@KRBTEST.COM
I20260928 08:44:39.106194 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:39.074018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35653 (local address 127.0.97.60:37393)
0928 08:44:39.076053 (+ 2035us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:39.076062 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:39.083370 (+ 7308us) server_negotiation.cc:407] Connection header received
0928 08:44:39.083466 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:39.083474 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:39.083595 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:39.083724 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:39.085316 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.087365 (+ 2049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.089772 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.090640 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.090765 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:39.098169 (+ 7404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:39.098175 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:39.098180 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:39.098221 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:39.102380 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:39.103620 (+ 1240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:39.103626 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:39.103630 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:39.103766 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:39.104273 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:39.104279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:39.104283 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:39.104606 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:39.104723 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:39.105620 (+ 897us) server_negotiation.cc:299] Negotiation successful
0928 08:44:39.105840 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1870,"thread_start_us":1722,"threads_started":1}
I20260928 08:44:39.109400 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:39.073563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37393 (local address 127.0.97.3:35653)
0928 08:44:39.083217 (+ 9654us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:39.083234 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:39.083367 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:39.083973 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:39.083978 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:39.084645 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:39.085112 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:39.085128 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.087593 (+ 2465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.087602 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:39.089604 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:39.089621 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.093578 (+ 3957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.093584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:39.093771 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:39.095342 (+ 1571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:39.095364 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:39.097950 (+ 2586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:39.102533 (+ 4583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:39.102538 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:39.102542 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:39.103130 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:39.103927 (+ 797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:39.103931 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:39.103934 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:39.104151 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:39.104807 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:39.104814 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:39.105213 (+ 399us) client_negotiation.cc:769] Sending connection context
0928 08:44:39.107450 (+ 2237us) client_negotiation.cc:240] Negotiation successful
0928 08:44:39.107994 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9550,"thread_start_us":8736,"threads_started":1}
I20260928 08:44:39.111155 7810 heartbeater.cc:347] Connected to a master server at 127.0.97.60:37393
I20260928 08:44:39.111490 7810 heartbeater.cc:464] Registering TS with master...
I20260928 08:44:39.112394 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:39.067459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38943 (local address 127.0.97.62:46083)
0928 08:44:39.084343 (+ 16884us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:39.084350 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:39.084369 (+ 19us) server_negotiation.cc:407] Connection header received
0928 08:44:39.084423 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:39.084427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:39.084495 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:39.085706 (+ 1211us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:39.085858 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.086891 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.088852 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.089434 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.089595 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:39.094066 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:39.094073 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:39.094079 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:39.094116 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:39.106155 (+ 12039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:39.108673 (+ 2518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:39.108679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:39.108682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:39.108854 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:39.109933 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:39.109940 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:39.109946 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:39.110340 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:39.110566 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:39.110981 (+ 415us) server_negotiation.cc:299] Negotiation successful
0928 08:44:39.111261 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16760,"thread_start_us":16667,"threads_started":1}
I20260928 08:44:39.112396 7810 heartbeater.cc:511] Master 127.0.97.60:37393 requested a full tablet report, sending...
I20260928 08:44:39.113879 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:39.067434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55059 (local address 127.0.97.61:36547)
0928 08:44:39.079835 (+ 12401us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:39.079842 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:39.079864 (+ 22us) server_negotiation.cc:407] Connection header received
0928 08:44:39.081332 (+ 1468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:39.081337 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:39.081424 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:39.081527 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:39.084004 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.085176 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.087709 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.088392 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.088499 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:39.094507 (+ 6008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:39.094513 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:39.094518 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:39.094563 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:39.098512 (+ 3949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:39.103888 (+ 5376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:39.103893 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:39.103897 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:39.104068 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:39.111305 (+ 7237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:39.111310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:39.111313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:39.111737 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:39.111896 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:39.113227 (+ 1331us) server_negotiation.cc:299] Negotiation successful
0928 08:44:39.113471 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12244,"thread_start_us":12429,"threads_started":1}
I20260928 08:44:39.114686 7341 ts_manager.cc:194] Registered new tserver with Master: be8863ce64e24035aaeed4bf282343b3 (127.0.97.3:41011)
I20260928 08:44:39.117312 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:39.068214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.97.3:38943)
0928 08:44:39.082625 (+ 14411us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:39.082642 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:39.082750 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:39.084654 (+ 1904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:39.084659 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:39.085232 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:39.085566 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:39.085576 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.087039 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.087045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:39.088678 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:39.088689 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.089640 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.089646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:39.089817 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:39.090313 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:39.090331 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:39.093862 (+ 3531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:39.106515 (+ 12653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:39.106522 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:39.106525 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:39.107062 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:39.109620 (+ 2558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:39.109626 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:39.109632 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:39.109822 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:39.110537 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:39.110542 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:39.110687 (+ 145us) client_negotiation.cc:769] Sending connection context
0928 08:44:39.110843 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 08:44:39.111101 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14231,"mutex_wait_us":3,"thread_start_us":13004,"threads_started":1}
I20260928 08:44:39.118361 7811 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46083
I20260928 08:44:39.118479 7811 heartbeater.cc:464] Registering TS with master...
I20260928 08:44:39.118964 7811 heartbeater.cc:511] Master 127.0.97.62:46083 requested a full tablet report, sending...
I20260928 08:44:39.120234 6552 ts_manager.cc:194] Re-registered known tserver with Master: be8863ce64e24035aaeed4bf282343b3 (127.0.97.3:41011)
I20260928 08:44:39.120637 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:39.068213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36547 (local address 127.0.97.3:55059)
0928 08:44:39.078717 (+ 10504us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:39.078786 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:39.080980 (+ 2194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:39.081950 (+ 970us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:39.081961 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:39.082815 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:39.083759 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:39.083789 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.085386 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.085393 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:39.087537 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:39.087560 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.088861 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.088868 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:39.089135 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:39.090173 (+ 1038us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:39.090219 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:39.094308 (+ 4089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:39.099021 (+ 4713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:39.099029 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:39.099045 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:39.103488 (+ 4443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:39.109953 (+ 6465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:39.109957 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:39.109962 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:39.110117 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:39.111986 (+ 1869us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:39.111990 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:39.112145 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 08:44:39.114369 (+ 2224us) client_negotiation.cc:240] Negotiation successful
0928 08:44:39.114660 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10105,"thread_start_us":14625,"threads_started":1}
I20260928 08:44:39.121359 6552 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:38943
I20260928 08:44:39.122162 7812 heartbeater.cc:347] Connected to a master server at 127.0.97.61:36547
I20260928 08:44:39.122262 7812 heartbeater.cc:464] Registering TS with master...
I20260928 08:44:39.122609 7812 heartbeater.cc:511] Master 127.0.97.61:36547 requested a full tablet report, sending...
I20260928 08:44:39.123593 7150 ts_manager.cc:194] Registered new tserver with Master: be8863ce64e24035aaeed4bf282343b3 (127.0.97.3:41011)
I20260928 08:44:39.133153 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:39.094349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48844 (local address 127.0.97.61:36547)
0928 08:44:39.114330 (+ 19981us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:39.114334 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:39.114412 (+ 78us) server_negotiation.cc:407] Connection header received
0928 08:44:39.114692 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:39.114700 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:39.114788 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:39.114874 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:39.117963 (+ 3089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.119117 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.121606 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.122307 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.122397 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:39.126534 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:39.126541 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:39.126545 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:39.126586 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:39.130429 (+ 3843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:39.131233 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:39.131237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:39.131240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:39.131386 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:39.131754 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:39.131758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:39.131762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:39.132145 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:39.132382 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:39.132718 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 08:44:39.132863 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19878,"thread_start_us":21299,"threads_started":1}
I20260928 08:44:39.135134 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:39.093768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35980 (local address 127.0.97.62:46083)
0928 08:44:39.112977 (+ 19209us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:39.112982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:39.116515 (+ 3533us) server_negotiation.cc:407] Connection header received
0928 08:44:39.116665 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:39.116671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:39.116746 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:39.116929 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:39.119680 (+ 2751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.121426 (+ 1746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.123186 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.123885 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.123985 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:39.127546 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:39.127552 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:39.127558 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:39.127617 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:39.131231 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:39.132024 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:39.132029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:39.132032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:39.132161 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:39.133122 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:39.133127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:39.133132 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:39.133468 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:39.133699 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:39.134602 (+ 903us) server_negotiation.cc:299] Negotiation successful
0928 08:44:39.134811 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19072,"thread_start_us":20014,"threads_started":1}
I20260928 08:44:39.135588 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:39.112384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40204 (local address 127.0.97.60:37393)
0928 08:44:39.112575 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:39.112583 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:39.117141 (+ 4558us) server_negotiation.cc:407] Connection header received
0928 08:44:39.117252 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:39.117258 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:39.117339 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:39.117435 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:39.122885 (+ 5450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.123920 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.125945 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.126515 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.126606 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:39.129266 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:39.129271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:39.129276 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:39.129313 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:39.132314 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:39.133515 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:39.133520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:39.133524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:39.133678 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:39.134026 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:39.134031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:39.134035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:39.134395 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:39.134619 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:39.135085 (+ 466us) server_negotiation.cc:299] Negotiation successful
0928 08:44:39.135280 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 08:44:39.138864 6552 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:44:39.146183 6552 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:44:39.197564 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:39.165164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53076 (local address 127.0.97.2:42561)
0928 08:44:39.190084 (+ 24920us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:39.190091 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:39.190111 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:44:39.190184 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:39.190189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:39.190373 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:39.191197 (+ 824us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:39.191350 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.192393 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.195150 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.196565 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.196698 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:39.196916 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:39.197137 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 08:44:39.197278 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24787,"thread_start_us":24682,"threads_started":1}
I20260928 08:44:39.197721 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:39.164958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42561 (local address 127.0.0.1:53076)
0928 08:44:39.188902 (+ 23944us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:39.188921 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:39.189043 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:39.190576 (+ 1533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:39.190580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:39.190616 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:39.191010 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:39.191020 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.192521 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.192529 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:39.194976 (+ 2447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:39.194990 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.196749 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.196756 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:39.196940 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:39.196973 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 08:44:39.197091 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 08:44:39.197221 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23801,"mutex_wait_us":32,"thread_start_us":15778,"threads_started":1}
I20260928 08:44:39.198593 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:39.165797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57568 (local address 127.0.97.1:46847)
0928 08:44:39.188424 (+ 22627us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:39.188438 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:39.189628 (+ 1190us) server_negotiation.cc:407] Connection header received
0928 08:44:39.189804 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:39.189816 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:39.190191 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:39.190482 (+ 291us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:39.191173 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.193050 (+ 1877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.195800 (+ 2750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.197222 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.197364 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:39.197587 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:39.198169 (+ 582us) server_negotiation.cc:299] Negotiation successful
0928 08:44:39.198335 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22392,"thread_start_us":22227,"threads_started":1}
I20260928 08:44:39.198683 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:39.165285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46847 (local address 127.0.0.1:57568)
0928 08:44:39.189488 (+ 24203us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:39.189504 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:39.189636 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:39.190468 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:39.190475 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:39.190505 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:39.190948 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:39.190963 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.193222 (+ 2259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.193230 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:39.195634 (+ 2404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:39.195646 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.197773 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.197779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:39.197973 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:39.198015 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 08:44:39.198123 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 08:44:39.198262 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24084,"mutex_wait_us":35,"thread_start_us":23351,"threads_started":1}
I20260928 08:44:39.202514 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:39.164662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36824 (local address 127.0.97.3:41011)
0928 08:44:39.193004 (+ 28342us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:39.193015 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:39.193045 (+ 30us) server_negotiation.cc:407] Connection header received
0928 08:44:39.193150 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:39.193157 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:39.193370 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:39.193569 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:39.194622 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.196280 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.199463 (+ 3183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.201461 (+ 1998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.201585 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:39.201766 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:39.202079 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 08:44:39.202224 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28184,"thread_start_us":28002,"threads_started":1}
I20260928 08:44:39.202540 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:39.164401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41011 (local address 127.0.0.1:36824)
0928 08:44:39.180421 (+ 16020us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:39.180448 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:39.180613 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:39.193582 (+ 12969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:39.193587 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:39.193616 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:39.194109 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:39.194122 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.196434 (+ 2312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.196439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:39.199296 (+ 2857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:39.199311 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.201664 (+ 2353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.201671 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:39.201864 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:39.201903 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 08:44:39.202004 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 08:44:39.202142 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15851,"mutex_wait_us":4,"thread_start_us":25184,"threads_started":1}
I20260928 08:44:39.202780 7604 tablet_service.cc:1467] Processing CreateTablet for tablet fde156e69b854614ad3fc5a53a83589b (DEFAULT_TABLE table=test-table [id=463ffe24477d4ed08533cac36f84def5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:39.202790 7455 tablet_service.cc:1467] Processing CreateTablet for tablet fde156e69b854614ad3fc5a53a83589b (DEFAULT_TABLE table=test-table [id=463ffe24477d4ed08533cac36f84def5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:39.203872 7604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fde156e69b854614ad3fc5a53a83589b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:39.203897 7455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fde156e69b854614ad3fc5a53a83589b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:39.210019 7745 tablet_service.cc:1467] Processing CreateTablet for tablet fde156e69b854614ad3fc5a53a83589b (DEFAULT_TABLE table=test-table [id=463ffe24477d4ed08533cac36f84def5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:39.211412 7745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fde156e69b854614ad3fc5a53a83589b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:39.217341 7843 tablet_bootstrap.cc:492] T fde156e69b854614ad3fc5a53a83589b P fb125058c6c34aae8dff092e2f334d3b: Bootstrap starting.
I20260928 08:44:39.218858 7844 tablet_bootstrap.cc:492] T fde156e69b854614ad3fc5a53a83589b P 1137082ae436457c84ba8100f189c4ec: Bootstrap starting.
I20260928 08:44:39.220867 7843 tablet_bootstrap.cc:654] T fde156e69b854614ad3fc5a53a83589b P fb125058c6c34aae8dff092e2f334d3b: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:39.222168 7843 log.cc:824] T fde156e69b854614ad3fc5a53a83589b P fb125058c6c34aae8dff092e2f334d3b: Log is configured to *not* fsync() on all Append() calls
I20260928 08:44:39.222445 7844 tablet_bootstrap.cc:654] T fde156e69b854614ad3fc5a53a83589b P 1137082ae436457c84ba8100f189c4ec: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:39.223978 7844 log.cc:824] T fde156e69b854614ad3fc5a53a83589b P 1137082ae436457c84ba8100f189c4ec: Log is configured to *not* fsync() on all Append() calls
I20260928 08:44:39.225046 7846 tablet_bootstrap.cc:492] T fde156e69b854614ad3fc5a53a83589b P be8863ce64e24035aaeed4bf282343b3: Bootstrap starting.
I20260928 08:44:39.225986 7843 tablet_bootstrap.cc:492] T fde156e69b854614ad3fc5a53a83589b P fb125058c6c34aae8dff092e2f334d3b: No bootstrap required, opened a new log
I20260928 08:44:39.226254 7843 ts_tablet_manager.cc:1397] T fde156e69b854614ad3fc5a53a83589b P fb125058c6c34aae8dff092e2f334d3b: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260928 08:44:39.228154 7844 tablet_bootstrap.cc:492] T fde156e69b854614ad3fc5a53a83589b P 1137082ae436457c84ba8100f189c4ec: No bootstrap required, opened a new log
I20260928 08:44:39.228288 7846 tablet_bootstrap.cc:654] T fde156e69b854614ad3fc5a53a83589b P be8863ce64e24035aaeed4bf282343b3: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:39.228410 7844 ts_tablet_manager.cc:1397] T fde156e69b854614ad3fc5a53a83589b P 1137082ae436457c84ba8100f189c4ec: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20260928 08:44:39.229386 7846 log.cc:824] T fde156e69b854614ad3fc5a53a83589b P be8863ce64e24035aaeed4bf282343b3: Log is configured to *not* fsync() on all Append() calls
I20260928 08:44:39.232717 7846 tablet_bootstrap.cc:492] T fde156e69b854614ad3fc5a53a83589b P be8863ce64e24035aaeed4bf282343b3: No bootstrap required, opened a new log
I20260928 08:44:39.232955 7846 ts_tablet_manager.cc:1397] T fde156e69b854614ad3fc5a53a83589b P be8863ce64e24035aaeed4bf282343b3: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 08:44:39.234721 7843 raft_consensus.cc:348] T fde156e69b854614ad3fc5a53a83589b P fb125058c6c34aae8dff092e2f334d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8863ce64e24035aaeed4bf282343b3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41011 } } peers { permanent_uuid: "1137082ae436457c84ba8100f189c4ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42561 } } peers { permanent_uuid: "fb125058c6c34aae8dff092e2f334d3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46847 } }
I20260928 08:44:39.235113 7843 raft_consensus.cc:374] T fde156e69b854614ad3fc5a53a83589b P fb125058c6c34aae8dff092e2f334d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:39.235255 7843 raft_consensus.cc:729] T fde156e69b854614ad3fc5a53a83589b P fb125058c6c34aae8dff092e2f334d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb125058c6c34aae8dff092e2f334d3b, State: Initialized, Role: FOLLOWER
I20260928 08:44:39.235984 7843 consensus_queue.cc:260] T fde156e69b854614ad3fc5a53a83589b P fb125058c6c34aae8dff092e2f334d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8863ce64e24035aaeed4bf282343b3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41011 } } peers { permanent_uuid: "1137082ae436457c84ba8100f189c4ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42561 } } peers { permanent_uuid: "fb125058c6c34aae8dff092e2f334d3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46847 } }
I20260928 08:44:39.238265 7843 ts_tablet_manager.cc:1428] T fde156e69b854614ad3fc5a53a83589b P fb125058c6c34aae8dff092e2f334d3b: Time spent starting tablet: real 0.012s user 0.005s sys 0.008s
I20260928 08:44:39.238211 7844 raft_consensus.cc:348] T fde156e69b854614ad3fc5a53a83589b P 1137082ae436457c84ba8100f189c4ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8863ce64e24035aaeed4bf282343b3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41011 } } peers { permanent_uuid: "1137082ae436457c84ba8100f189c4ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42561 } } peers { permanent_uuid: "fb125058c6c34aae8dff092e2f334d3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46847 } }
I20260928 08:44:39.238587 7844 raft_consensus.cc:374] T fde156e69b854614ad3fc5a53a83589b P 1137082ae436457c84ba8100f189c4ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:39.238687 7844 raft_consensus.cc:729] T fde156e69b854614ad3fc5a53a83589b P 1137082ae436457c84ba8100f189c4ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1137082ae436457c84ba8100f189c4ec, State: Initialized, Role: FOLLOWER
I20260928 08:44:39.239267 7844 consensus_queue.cc:260] T fde156e69b854614ad3fc5a53a83589b P 1137082ae436457c84ba8100f189c4ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8863ce64e24035aaeed4bf282343b3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41011 } } peers { permanent_uuid: "1137082ae436457c84ba8100f189c4ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42561 } } peers { permanent_uuid: "fb125058c6c34aae8dff092e2f334d3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46847 } }
I20260928 08:44:39.243011 7670 heartbeater.cc:503] Master 127.0.97.62:46083 was elected leader, sending a full tablet report...
I20260928 08:44:39.243672 7844 ts_tablet_manager.cc:1428] T fde156e69b854614ad3fc5a53a83589b P 1137082ae436457c84ba8100f189c4ec: Time spent starting tablet: real 0.015s user 0.008s sys 0.007s
I20260928 08:44:39.248245 7846 raft_consensus.cc:348] T fde156e69b854614ad3fc5a53a83589b P be8863ce64e24035aaeed4bf282343b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8863ce64e24035aaeed4bf282343b3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41011 } } peers { permanent_uuid: "1137082ae436457c84ba8100f189c4ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42561 } } peers { permanent_uuid: "fb125058c6c34aae8dff092e2f334d3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46847 } }
I20260928 08:44:39.248594 7846 raft_consensus.cc:374] T fde156e69b854614ad3fc5a53a83589b P be8863ce64e24035aaeed4bf282343b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:39.248709 7846 raft_consensus.cc:729] T fde156e69b854614ad3fc5a53a83589b P be8863ce64e24035aaeed4bf282343b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be8863ce64e24035aaeed4bf282343b3, State: Initialized, Role: FOLLOWER
I20260928 08:44:39.249224 7846 consensus_queue.cc:260] T fde156e69b854614ad3fc5a53a83589b P be8863ce64e24035aaeed4bf282343b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8863ce64e24035aaeed4bf282343b3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41011 } } peers { permanent_uuid: "1137082ae436457c84ba8100f189c4ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42561 } } peers { permanent_uuid: "fb125058c6c34aae8dff092e2f334d3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46847 } }
I20260928 08:44:39.250207 7811 heartbeater.cc:503] Master 127.0.97.62:46083 was elected leader, sending a full tablet report...
I20260928 08:44:39.250787 7846 ts_tablet_manager.cc:1428] T fde156e69b854614ad3fc5a53a83589b P be8863ce64e24035aaeed4bf282343b3: Time spent starting tablet: real 0.018s user 0.013s sys 0.000s
W20260928 08:44:39.271282 7672 tablet.cc:2369] T fde156e69b854614ad3fc5a53a83589b P 1137082ae436457c84ba8100f189c4ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:44:39.334125 7813 tablet.cc:2369] T fde156e69b854614ad3fc5a53a83589b P be8863ce64e24035aaeed4bf282343b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:44:39.393803 7528 tablet.cc:2369] T fde156e69b854614ad3fc5a53a83589b P fb125058c6c34aae8dff092e2f334d3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:44:39.408542 7849 raft_consensus.cc:482] T fde156e69b854614ad3fc5a53a83589b P fb125058c6c34aae8dff092e2f334d3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:44:39.408866 7849 raft_consensus.cc:504] T fde156e69b854614ad3fc5a53a83589b P fb125058c6c34aae8dff092e2f334d3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8863ce64e24035aaeed4bf282343b3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41011 } } peers { permanent_uuid: "1137082ae436457c84ba8100f189c4ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42561 } } peers { permanent_uuid: "fb125058c6c34aae8dff092e2f334d3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46847 } }
I20260928 08:44:39.410696 7849 leader_election.cc:288] T fde156e69b854614ad3fc5a53a83589b P fb125058c6c34aae8dff092e2f334d3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be8863ce64e24035aaeed4bf282343b3 (127.0.97.3:41011), 1137082ae436457c84ba8100f189c4ec (127.0.97.2:42561)
I20260928 08:44:39.420220 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:39.411013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42561 (local address 127.0.97.1:58137)
0928 08:44:39.412447 (+ 1434us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:39.412466 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:39.412604 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:39.413111 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:39.413115 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:39.413137 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:39.413498 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:39.413531 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.415082 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.415088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:39.417873 (+ 2785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:39.417886 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.419519 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.419524 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:39.419636 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:39.419660 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 08:44:39.419736 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 08:44:39.419918 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1284,"mutex_wait_us":5,"thread_start_us":1016,"threads_started":1}
I20260928 08:44:39.420208 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:39.411131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58137 (local address 127.0.97.2:42561)
0928 08:44:39.411324 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:39.411332 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:39.412542 (+ 1210us) server_negotiation.cc:407] Connection header received
0928 08:44:39.412777 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:39.412784 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:39.412861 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:39.412949 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:39.413708 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.414920 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.418031 (+ 3111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.419345 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.419446 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:39.419516 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:39.419788 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 08:44:39.419964 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 08:44:39.421507 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:39.411028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53475 (local address 127.0.97.3:41011)
0928 08:44:39.411246 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:39.411253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:39.412010 (+ 757us) server_negotiation.cc:407] Connection header received
0928 08:44:39.412354 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:39.412358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:39.412456 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:39.412537 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:39.413307 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.414617 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.419582 (+ 4965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.420728 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.420847 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:39.420915 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:39.421273 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 08:44:39.421357 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 08:44:39.421319 7624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde156e69b854614ad3fc5a53a83589b" candidate_uuid: "fb125058c6c34aae8dff092e2f334d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1137082ae436457c84ba8100f189c4ec" is_pre_election: true
I20260928 08:44:39.421739 7852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:39.410861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41011 (local address 127.0.97.1:53475)
0928 08:44:39.411888 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:39.411914 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:39.412136 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:39.412749 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:39.412753 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:39.412786 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:39.413110 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:39.413120 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.414775 (+ 1655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.414780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:39.417882 (+ 3102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:39.417891 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.420935 (+ 3044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.420941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:39.421111 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:39.421147 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 08:44:39.421248 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 08:44:39.421395 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":806,"mutex_wait_us":92,"thread_start_us":598,"threads_started":1}
I20260928 08:44:39.421887 7624 raft_consensus.cc:2454] T fde156e69b854614ad3fc5a53a83589b P 1137082ae436457c84ba8100f189c4ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb125058c6c34aae8dff092e2f334d3b in term 0.
I20260928 08:44:39.422556 7409 leader_election.cc:302] T fde156e69b854614ad3fc5a53a83589b P fb125058c6c34aae8dff092e2f334d3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1137082ae436457c84ba8100f189c4ec, fb125058c6c34aae8dff092e2f334d3b; no voters:
I20260928 08:44:39.422689 7765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde156e69b854614ad3fc5a53a83589b" candidate_uuid: "fb125058c6c34aae8dff092e2f334d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be8863ce64e24035aaeed4bf282343b3" is_pre_election: true
I20260928 08:44:39.423012 7849 raft_consensus.cc:2781] T fde156e69b854614ad3fc5a53a83589b P fb125058c6c34aae8dff092e2f334d3b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:44:39.423163 7849 raft_consensus.cc:482] T fde156e69b854614ad3fc5a53a83589b P fb125058c6c34aae8dff092e2f334d3b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:44:39.423154 7765 raft_consensus.cc:2454] T fde156e69b854614ad3fc5a53a83589b P be8863ce64e24035aaeed4bf282343b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb125058c6c34aae8dff092e2f334d3b in term 0.
I20260928 08:44:39.423293 7849 raft_consensus.cc:3037] T fde156e69b854614ad3fc5a53a83589b P fb125058c6c34aae8dff092e2f334d3b [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:39.425498 7849 raft_consensus.cc:504] T fde156e69b854614ad3fc5a53a83589b P fb125058c6c34aae8dff092e2f334d3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8863ce64e24035aaeed4bf282343b3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41011 } } peers { permanent_uuid: "1137082ae436457c84ba8100f189c4ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42561 } } peers { permanent_uuid: "fb125058c6c34aae8dff092e2f334d3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46847 } }
I20260928 08:44:39.426118 7849 leader_election.cc:288] T fde156e69b854614ad3fc5a53a83589b P fb125058c6c34aae8dff092e2f334d3b [CANDIDATE]: Term 1 election: Requested vote from peers be8863ce64e24035aaeed4bf282343b3 (127.0.97.3:41011), 1137082ae436457c84ba8100f189c4ec (127.0.97.2:42561)
I20260928 08:44:39.426544 7765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde156e69b854614ad3fc5a53a83589b" candidate_uuid: "fb125058c6c34aae8dff092e2f334d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be8863ce64e24035aaeed4bf282343b3"
I20260928 08:44:39.426623 7624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde156e69b854614ad3fc5a53a83589b" candidate_uuid: "fb125058c6c34aae8dff092e2f334d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1137082ae436457c84ba8100f189c4ec"
I20260928 08:44:39.426746 7765 raft_consensus.cc:3037] T fde156e69b854614ad3fc5a53a83589b P be8863ce64e24035aaeed4bf282343b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:39.426856 7624 raft_consensus.cc:3037] T fde156e69b854614ad3fc5a53a83589b P 1137082ae436457c84ba8100f189c4ec [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:39.430018 7624 raft_consensus.cc:2454] T fde156e69b854614ad3fc5a53a83589b P 1137082ae436457c84ba8100f189c4ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb125058c6c34aae8dff092e2f334d3b in term 1.
I20260928 08:44:39.430018 7765 raft_consensus.cc:2454] T fde156e69b854614ad3fc5a53a83589b P be8863ce64e24035aaeed4bf282343b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb125058c6c34aae8dff092e2f334d3b in term 1.
I20260928 08:44:39.430403 7409 leader_election.cc:302] T fde156e69b854614ad3fc5a53a83589b P fb125058c6c34aae8dff092e2f334d3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1137082ae436457c84ba8100f189c4ec, fb125058c6c34aae8dff092e2f334d3b; no voters:
I20260928 08:44:39.430707 7849 raft_consensus.cc:2781] T fde156e69b854614ad3fc5a53a83589b P fb125058c6c34aae8dff092e2f334d3b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:44:39.431532 7849 raft_consensus.cc:686] T fde156e69b854614ad3fc5a53a83589b P fb125058c6c34aae8dff092e2f334d3b [term 1 LEADER]: Becoming Leader. State: Replica: fb125058c6c34aae8dff092e2f334d3b, State: Running, Role: LEADER
I20260928 08:44:39.432081 7849 consensus_queue.cc:237] T fde156e69b854614ad3fc5a53a83589b P fb125058c6c34aae8dff092e2f334d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8863ce64e24035aaeed4bf282343b3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41011 } } peers { permanent_uuid: "1137082ae436457c84ba8100f189c4ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42561 } } peers { permanent_uuid: "fb125058c6c34aae8dff092e2f334d3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46847 } }
I20260928 08:44:39.436928 6551 catalog_manager.cc:5495] T fde156e69b854614ad3fc5a53a83589b P fb125058c6c34aae8dff092e2f334d3b reported cstate change: term changed from 0 to 1, leader changed from <none> to fb125058c6c34aae8dff092e2f334d3b (127.0.97.1). New cstate: current_term: 1 leader_uuid: "fb125058c6c34aae8dff092e2f334d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8863ce64e24035aaeed4bf282343b3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1137082ae436457c84ba8100f189c4ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb125058c6c34aae8dff092e2f334d3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46847 } health_report { overall_health: HEALTHY } } }
I20260928 08:44:39.485831 7842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:39.481123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57580 (local address 127.0.97.1:46847)
0928 08:44:39.481293 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:39.481298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:39.481314 (+ 16us) server_negotiation.cc:407] Connection header received
0928 08:44:39.481392 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:39.481395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:39.481471 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:39.481589 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:44:39.481923 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.482585 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.483843 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.484211 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.484302 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:39.484708 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:44:39.485207 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:44:39.485313 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:39.485440 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 08:44:39.485538 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 08:44:39.494028 7624 raft_consensus.cc:1261] T fde156e69b854614ad3fc5a53a83589b P 1137082ae436457c84ba8100f189c4ec [term 1 FOLLOWER]: Refusing update from remote peer fb125058c6c34aae8dff092e2f334d3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:44:39.494030 7765 raft_consensus.cc:1261] T fde156e69b854614ad3fc5a53a83589b P be8863ce64e24035aaeed4bf282343b3 [term 1 FOLLOWER]: Refusing update from remote peer fb125058c6c34aae8dff092e2f334d3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:44:39.494973 7849 consensus_queue.cc:1035] T fde156e69b854614ad3fc5a53a83589b P fb125058c6c34aae8dff092e2f334d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "be8863ce64e24035aaeed4bf282343b3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:44:39.495244 7854 consensus_queue.cc:1035] T fde156e69b854614ad3fc5a53a83589b P fb125058c6c34aae8dff092e2f334d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1137082ae436457c84ba8100f189c4ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:44:39.540563 6551 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:44:39.540891 6551 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:44:39.548257 6551 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0c813b2df2b645b18971acadaa4ca63a: Sending DeleteTablet for 3 replicas of tablet fde156e69b854614ad3fc5a53a83589b
I20260928 08:44:39.549320 7604 tablet_service.cc:1514] Processing DeleteTablet for tablet fde156e69b854614ad3fc5a53a83589b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:39 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53076
I20260928 08:44:39.549540 7744 tablet_service.cc:1514] Processing DeleteTablet for tablet fde156e69b854614ad3fc5a53a83589b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:39 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36824
I20260928 08:44:39.549907 7745 tablet_service.cc:1514] Processing DeleteTablet for tablet fde156e69b854614ad3fc5a53a83589b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:39 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36824
I20260928 08:44:39.550217 7866 tablet_replica.cc:330] T fde156e69b854614ad3fc5a53a83589b P 1137082ae436457c84ba8100f189c4ec: stopping tablet replica
I20260928 08:44:39.550733 7866 raft_consensus.cc:2229] T fde156e69b854614ad3fc5a53a83589b P 1137082ae436457c84ba8100f189c4ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:44:39.551280 7866 raft_consensus.cc:2258] T fde156e69b854614ad3fc5a53a83589b P 1137082ae436457c84ba8100f189c4ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:39.551467 7455 tablet_service.cc:1514] Processing DeleteTablet for tablet fde156e69b854614ad3fc5a53a83589b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:39 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57568
I20260928 08:44:39.553594 7866 ts_tablet_manager.cc:1905] T fde156e69b854614ad3fc5a53a83589b P 1137082ae436457c84ba8100f189c4ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:39.556288 7868 tablet_replica.cc:330] T fde156e69b854614ad3fc5a53a83589b P fb125058c6c34aae8dff092e2f334d3b: stopping tablet replica
I20260928 08:44:39.556757 7868 raft_consensus.cc:2229] T fde156e69b854614ad3fc5a53a83589b P fb125058c6c34aae8dff092e2f334d3b [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:44:39.557464 7868 raft_consensus.cc:2258] T fde156e69b854614ad3fc5a53a83589b P fb125058c6c34aae8dff092e2f334d3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:39.557986 7867 tablet_replica.cc:330] T fde156e69b854614ad3fc5a53a83589b P be8863ce64e24035aaeed4bf282343b3: stopping tablet replica
I20260928 08:44:39.558372 7867 raft_consensus.cc:2229] T fde156e69b854614ad3fc5a53a83589b P be8863ce64e24035aaeed4bf282343b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:44:39.558809 7867 raft_consensus.cc:2258] T fde156e69b854614ad3fc5a53a83589b P be8863ce64e24035aaeed4bf282343b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:39.560165 7868 ts_tablet_manager.cc:1905] T fde156e69b854614ad3fc5a53a83589b P fb125058c6c34aae8dff092e2f334d3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:39.563354 7866 ts_tablet_manager.cc:1918] T fde156e69b854614ad3fc5a53a83589b P 1137082ae436457c84ba8100f189c4ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:44:39.563508 7866 log.cc:1196] T fde156e69b854614ad3fc5a53a83589b P 1137082ae436457c84ba8100f189c4ec: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-1/wal/wals/fde156e69b854614ad3fc5a53a83589b
I20260928 08:44:39.564131 7866 ts_tablet_manager.cc:1939] T fde156e69b854614ad3fc5a53a83589b P 1137082ae436457c84ba8100f189c4ec: Deleting consensus metadata
I20260928 08:44:39.565733 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 6520
I20260928 08:44:39.566363 7867 ts_tablet_manager.cc:1905] T fde156e69b854614ad3fc5a53a83589b P be8863ce64e24035aaeed4bf282343b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:39.566843 6582 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:39.571056 7868 ts_tablet_manager.cc:1918] T fde156e69b854614ad3fc5a53a83589b P fb125058c6c34aae8dff092e2f334d3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:44:39.571225 7868 log.cc:1196] T fde156e69b854614ad3fc5a53a83589b P fb125058c6c34aae8dff092e2f334d3b: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-0/wal/wals/fde156e69b854614ad3fc5a53a83589b
I20260928 08:44:39.571762 7868 ts_tablet_manager.cc:1939] T fde156e69b854614ad3fc5a53a83589b P fb125058c6c34aae8dff092e2f334d3b: Deleting consensus metadata
I20260928 08:44:39.575248 7867 ts_tablet_manager.cc:1918] T fde156e69b854614ad3fc5a53a83589b P be8863ce64e24035aaeed4bf282343b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:44:39.575456 7867 log.cc:1196] T fde156e69b854614ad3fc5a53a83589b P be8863ce64e24035aaeed4bf282343b3: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-2/wal/wals/fde156e69b854614ad3fc5a53a83589b
I20260928 08:44:39.576076 7867 ts_tablet_manager.cc:1939] T fde156e69b854614ad3fc5a53a83589b P be8863ce64e24035aaeed4bf282343b3: Deleting consensus metadata
I20260928 08:44:39.576787 6539 catalog_manager.cc:4841] TS fb125058c6c34aae8dff092e2f334d3b (127.0.97.1:46847): tablet fde156e69b854614ad3fc5a53a83589b (table test-table [id=463ffe24477d4ed08533cac36f84def5]) successfully deleted
I20260928 08:44:39.585565 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:39.552933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48856 (local address 127.0.97.61:36547)
0928 08:44:39.553105 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:39.553113 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:39.564490 (+ 11377us) server_negotiation.cc:407] Connection header received
0928 08:44:39.564646 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:39.564651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:39.564728 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:39.564848 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:39.569997 (+ 5149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.571612 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.573507 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.574413 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.574602 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:39.578066 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:39.578076 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:39.578084 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:39.578138 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:39.582535 (+ 4397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:39.583183 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:39.583189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:39.583193 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:39.583331 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:39.583734 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:39.583739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:39.583742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:39.584108 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:39.584403 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:39.584977 (+ 574us) server_negotiation.cc:299] Negotiation successful
0928 08:44:39.585183 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 08:44:39.599030 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:39.564549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40208 (local address 127.0.97.60:37393)
0928 08:44:39.564716 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:39.564722 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:39.585301 (+ 20579us) server_negotiation.cc:407] Connection header received
0928 08:44:39.585443 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:39.585449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:39.585533 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:39.585638 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:39.587456 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.588643 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.589846 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.590501 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.590631 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:39.593166 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:39.593174 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:39.593179 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:39.593219 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:39.596063 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:39.596659 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:39.596671 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:39.596675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:39.596800 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:39.597569 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:39.597576 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:39.597579 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:39.597879 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:39.598177 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:39.598465 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 08:44:39.598661 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 08:44:39.743155 6537 catalog_manager.cc:4841] TS 1137082ae436457c84ba8100f189c4ec (127.0.97.2:42561): tablet fde156e69b854614ad3fc5a53a83589b (table test-table [id=463ffe24477d4ed08533cac36f84def5]) successfully deleted
I20260928 08:44:39.744019 6538 catalog_manager.cc:4841] TS be8863ce64e24035aaeed4bf282343b3 (127.0.97.3:41011): tablet fde156e69b854614ad3fc5a53a83589b (table test-table [id=463ffe24477d4ed08533cac36f84def5]) successfully deleted
W20260928 08:44:39.744303 6538 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.97.3:41011, user_credentials={real_user=slave}} blocked reactor thread for 166513us
W20260928 08:44:39.744814 6538 catalog_manager.cc:4808] TS be8863ce64e24035aaeed4bf282343b3 (127.0.97.3:41011): delete failed for tablet fde156e69b854614ad3fc5a53a83589b because the tablet was not found. No further retry: Not found: Tablet not found: fde156e69b854614ad3fc5a53a83589b
I20260928 08:44:39.745299 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 6520
I20260928 08:44:39.804965 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:39.778431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40216 (local address 127.0.97.60:37393)
0928 08:44:39.778964 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:39.778972 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:39.787852 (+ 8880us) server_negotiation.cc:407] Connection header received
0928 08:44:39.788285 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:39.788292 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:39.788377 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:39.788457 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:39.790753 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.792083 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.794068 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.794780 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.794898 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:39.798645 (+ 3747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:39.798651 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:39.798654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:39.798688 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:39.801706 (+ 3018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:39.802696 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:39.802702 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:39.802705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:39.802848 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:39.803260 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:39.803266 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:39.803270 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:39.803685 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:39.803963 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:39.804325 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 08:44:39.804553 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347}
I20260928 08:44:39.805835 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:39.779144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48860 (local address 127.0.97.61:36547)
0928 08:44:39.779368 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:39.779376 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:39.786537 (+ 7161us) server_negotiation.cc:407] Connection header received
0928 08:44:39.787127 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:39.787137 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:39.787256 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:39.787343 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:39.789691 (+ 2348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.790797 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.792894 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:39.793604 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:39.793713 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:39.798426 (+ 4713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:39.798433 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:39.798439 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:39.798480 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:39.801912 (+ 3432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:39.802791 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:39.802797 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:39.802801 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:39.802970 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:39.803362 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:39.803368 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:39.803371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:39.803844 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:39.804418 (+ 574us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:39.805235 (+ 817us) server_negotiation.cc:299] Negotiation successful
0928 08:44:39.805456 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 08:44:40.535037 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:40.534186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.97.1:55849)
0928 08:44:40.534647 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:40.534802 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":253,"threads_started":1}
W20260928 08:44:40.535444 7521 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46083 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
I20260928 08:44:40.536237 7879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:40.535399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.97.2:50785)
0928 08:44:40.535931 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:40.536039 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":284,"threads_started":1}
W20260928 08:44:40.536626 7670 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46083 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
I20260928 08:44:40.553509 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:40.552865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.97.3:38999)
0928 08:44:40.553238 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:40.553344 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":186,"threads_started":1}
W20260928 08:44:40.553879 7811 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46083 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
I20260928 08:44:41.047642 7881 raft_consensus.cc:482] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0c813b2df2b645b18971acadaa4ca63a)
I20260928 08:44:41.047931 7881 raft_consensus.cc:504] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: false } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37393 } attrs { promote: false } }
I20260928 08:44:41.048704 7882 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0c813b2df2b645b18971acadaa4ca63a)
I20260928 08:44:41.048827 7881 leader_election.cc:288] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c813b2df2b645b18971acadaa4ca63a (127.0.97.62:46083), 17003f6ce92c4415a110af3b597c8e4a (127.0.97.61:36547)
I20260928 08:44:41.049094 7882 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: false } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37393 } attrs { promote: false } }
I20260928 08:44:41.057474 7882 leader_election.cc:288] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0c813b2df2b645b18971acadaa4ca63a (127.0.97.62:46083), a63285bae4b64ecca10248bb821d965f (127.0.97.60:37393)
I20260928 08:44:41.060818 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.048982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.0.1:36426)
0928 08:44:41.060193 (+ 11211us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:41.060415 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11048,"threads_started":1}
W20260928 08:44:41.064028 7325 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
I20260928 08:44:41.064656 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.051388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.0.1:36428)
0928 08:44:41.064188 (+ 12800us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:41.064373 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12681,"threads_started":1}
W20260928 08:44:41.065256 7135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
I20260928 08:44:41.070370 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.049194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48862 (local address 127.0.97.61:36547)
0928 08:44:41.059776 (+ 10582us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:41.059784 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:41.061715 (+ 1931us) server_negotiation.cc:407] Connection header received
0928 08:44:41.061805 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:41.061813 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:41.061920 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:41.062022 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:41.062741 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.063924 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.066293 (+ 2369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.068376 (+ 2083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.068517 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:41.068612 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:41.069633 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0928 08:44:41.069879 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":10398,"thread_start_us":10198,"threads_started":1}
I20260928 08:44:41.071274 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.049209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:36547 (local address 127.0.0.1:48862)
0928 08:44:41.061367 (+ 12158us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:41.061389 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:41.061566 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:41.062246 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:41.062254 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:41.062290 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:41.062563 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:41.062574 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.064103 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.064108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:41.066124 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:41.066139 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.068739 (+ 2600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.068745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:41.069004 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:41.069086 (+ 82us) client_negotiation.cc:769] Sending connection context
0928 08:44:41.069181 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 08:44:41.069330 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12038,"thread_start_us":11778,"threads_started":1}
I20260928 08:44:41.071297 7890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.069456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.0.1:36442)
0928 08:44:41.070837 (+ 1381us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:41.071001 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":873,"thread_start_us":678,"threads_started":1}
I20260928 08:44:41.071358 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.070662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.0.1:36446)
0928 08:44:41.070847 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:41.070995 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 08:44:41.071655 7325 leader_election.cc:334] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0c813b2df2b645b18971acadaa4ca63a (127.0.97.62:46083): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
I20260928 08:44:41.072213 7160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a63285bae4b64ecca10248bb821d965f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "17003f6ce92c4415a110af3b597c8e4a" is_pre_election: true
W20260928 08:44:41.071491 7135 leader_election.cc:334] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0c813b2df2b645b18971acadaa4ca63a (127.0.97.62:46083): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
I20260928 08:44:41.072536 7160 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a63285bae4b64ecca10248bb821d965f in term 1.
I20260928 08:44:41.073112 7326 leader_election.cc:302] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17003f6ce92c4415a110af3b597c8e4a, a63285bae4b64ecca10248bb821d965f; no voters: 0c813b2df2b645b18971acadaa4ca63a
I20260928 08:44:41.073544 7881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 08:44:41.073674 7881 raft_consensus.cc:482] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0c813b2df2b645b18971acadaa4ca63a)
I20260928 08:44:41.073791 7881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [term 1 FOLLOWER]: Advancing to term 2
I20260928 08:44:41.076020 7886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.068086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:37393 (local address 127.0.0.1:40220)
0928 08:44:41.068260 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:41.068293 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:41.068471 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:41.069369 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:41.069373 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:41.069400 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:41.069856 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:41.069868 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.071471 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.071476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:41.073502 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:41.073515 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.075186 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.075193 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:41.075383 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:41.075419 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 08:44:41.075554 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 08:44:41.075736 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":47}
I20260928 08:44:41.076951 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.060006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40220 (local address 127.0.97.60:37393)
0928 08:44:41.068116 (+ 8110us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:41.068123 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:41.068607 (+ 484us) server_negotiation.cc:407] Connection header received
0928 08:44:41.068682 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:41.068694 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:41.068780 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:41.068885 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:41.070071 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.071311 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.073735 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.074941 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.075170 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:41.075308 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:41.076449 (+ 1141us) server_negotiation.cc:299] Negotiation successful
0928 08:44:41.076577 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7993,"thread_start_us":7875,"threads_started":1}
I20260928 08:44:41.076972 7881 raft_consensus.cc:504] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: false } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37393 } attrs { promote: false } }
I20260928 08:44:41.077525 7351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "17003f6ce92c4415a110af3b597c8e4a" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a63285bae4b64ecca10248bb821d965f" is_pre_election: true
I20260928 08:44:41.077951 7351 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 17003f6ce92c4415a110af3b597c8e4a in current term 2: Already voted for candidate a63285bae4b64ecca10248bb821d965f in this term.
I20260928 08:44:41.078262 7881 leader_election.cc:288] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [CANDIDATE]: Term 2 election: Requested vote from peers 0c813b2df2b645b18971acadaa4ca63a (127.0.97.62:46083), 17003f6ce92c4415a110af3b597c8e4a (127.0.97.61:36547)
I20260928 08:44:41.078375 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.077904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.0.1:36464)
0928 08:44:41.078119 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:41.078206 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260928 08:44:41.078430 7160 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a63285bae4b64ecca10248bb821d965f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "17003f6ce92c4415a110af3b597c8e4a"
I20260928 08:44:41.078622 7160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [term 1 FOLLOWER]: Advancing to term 2
I20260928 08:44:41.079265 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.078971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.0.1:36470)
0928 08:44:41.079087 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:41.079145 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 08:44:41.079499 7325 leader_election.cc:334] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0c813b2df2b645b18971acadaa4ca63a (127.0.97.62:46083): Network error: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
I20260928 08:44:41.080250 7137 leader_election.cc:302] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 17003f6ce92c4415a110af3b597c8e4a; no voters: 0c813b2df2b645b18971acadaa4ca63a, a63285bae4b64ecca10248bb821d965f
I20260928 08:44:41.081282 7160 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a63285bae4b64ecca10248bb821d965f in term 2.
I20260928 08:44:41.081462 7882 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 08:44:41.081815 7326 leader_election.cc:302] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17003f6ce92c4415a110af3b597c8e4a, a63285bae4b64ecca10248bb821d965f; no voters: 0c813b2df2b645b18971acadaa4ca63a
I20260928 08:44:41.082111 7881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [term 2 FOLLOWER]: Leader election won for term 2
I20260928 08:44:41.082626 7881 raft_consensus.cc:686] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [term 2 LEADER]: Becoming Leader. State: Replica: a63285bae4b64ecca10248bb821d965f, State: Running, Role: LEADER
I20260928 08:44:41.083071 7881 consensus_queue.cc:237] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: false } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37393 } attrs { promote: false } }
I20260928 08:44:41.084978 7891 sys_catalog.cc:455] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a63285bae4b64ecca10248bb821d965f. Latest consensus state: current_term: 2 leader_uuid: "a63285bae4b64ecca10248bb821d965f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: false } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37393 } attrs { promote: false } } }
I20260928 08:44:41.085305 7891 sys_catalog.cc:458] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:41.085929 7893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:44:41.089408 7893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:44:41.090205 7893 catalog_manager.cc:1259] Loaded cluster ID: cd79b6032ebe422ab25f6fc91d390632
I20260928 08:44:41.090276 7893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:44:41.091380 7893 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:44:41.092221 7893 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f: Loaded TSK: 0
I20260928 08:44:41.093045 7893 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:44:41.136855 7341 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40216:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:44:41.144539 7341 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:44:41.148666 7160 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [term 2 FOLLOWER]: Refusing update from remote peer a63285bae4b64ecca10248bb821d965f: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 08:44:41.148725 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.148288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.0.1:36486)
0928 08:44:41.148476 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:41.148588 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 08:44:41.149405 7891 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [LEADER]: Connected to new peer: Peer: permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 08:44:41.149763 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.149336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.0.1:36490)
0928 08:44:41.149508 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:41.149605 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 08:44:41.150017 7325 consensus_peers.cc:487] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f -> Peer 0c813b2df2b645b18971acadaa4ca63a (127.0.97.62:46083): Couldn't send request to peer 0c813b2df2b645b18971acadaa4ca63a. Status: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 08:44:41.152164 7882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a63285bae4b64ecca10248bb821d965f. Latest consensus state: current_term: 2 leader_uuid: "a63285bae4b64ecca10248bb821d965f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: false } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37393 } attrs { promote: false } } }
I20260928 08:44:41.152503 7882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:44:41.154053 7891 sys_catalog.cc:455] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a63285bae4b64ecca10248bb821d965f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: false } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37393 } attrs { promote: false } } }
I20260928 08:44:41.154341 7891 sys_catalog.cc:458] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:41.154827 7882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a63285bae4b64ecca10248bb821d965f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "0c813b2df2b645b18971acadaa4ca63a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46083 } } peers { permanent_uuid: "17003f6ce92c4415a110af3b597c8e4a" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 36547 } attrs { promote: false } } peers { permanent_uuid: "a63285bae4b64ecca10248bb821d965f" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 37393 } attrs { promote: false } } }
I20260928 08:44:41.155123 7882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17003f6ce92c4415a110af3b597c8e4a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:44:41.172909 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.162835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42561 (local address 127.0.0.1:53080)
0928 08:44:41.162967 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:41.162985 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:41.163077 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:41.164335 (+ 1258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:41.164343 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:41.164371 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:41.164572 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:41.164584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.166807 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.166815 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:41.169603 (+ 2788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:41.169622 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.171825 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.171835 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:41.172108 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:41.172161 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 08:44:41.172274 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 08:44:41.172478 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 08:44:41.174336 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.162989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53080 (local address 127.0.97.2:42561)
0928 08:44:41.163944 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:41.163950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:41.163971 (+ 21us) server_negotiation.cc:407] Connection header received
0928 08:44:41.164050 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:41.164058 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:41.164133 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:41.164740 (+ 607us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:41.164943 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.166646 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.169802 (+ 3156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.171622 (+ 1820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.171801 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:41.172027 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:41.172318 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 08:44:41.173858 (+ 1540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":796,"thread_start_us":686,"threads_started":1}
I20260928 08:44:41.175423 7604 tablet_service.cc:1467] Processing CreateTablet for tablet d3deecb68bf342b48ce6a312a11dcb11 (DEFAULT_TABLE table=test-table [id=b2f68156fd0340e69e0122c2352fe9ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:41.176170 7604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3deecb68bf342b48ce6a312a11dcb11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:41.182974 7883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.162394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:41011 (local address 127.0.0.1:36828)
0928 08:44:41.162522 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:41.162538 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:41.162660 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:41.178341 (+ 15681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:41.178346 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:41.178369 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:41.178622 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:41.178631 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.179676 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.179684 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:41.181149 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:41.181159 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.182293 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.182299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:41.182454 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:41.182475 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 08:44:41.182578 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 08:44:41.182698 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":7}
I20260928 08:44:41.183004 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.162750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36828 (local address 127.0.97.3:41011)
0928 08:44:41.177948 (+ 15198us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:41.177955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:41.177972 (+ 17us) server_negotiation.cc:407] Connection header received
0928 08:44:41.178033 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:41.178039 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:41.178105 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:41.178752 (+ 647us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:41.178947 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.179559 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.181307 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.182074 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.182209 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:41.182328 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:41.182602 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 08:44:41.182731 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15062,"thread_start_us":15426,"threads_started":1}
I20260928 08:44:41.184078 7903 tablet_bootstrap.cc:492] T d3deecb68bf342b48ce6a312a11dcb11 P 1137082ae436457c84ba8100f189c4ec: Bootstrap starting.
I20260928 08:44:41.184527 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.163756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46847 (local address 127.0.0.1:57590)
0928 08:44:41.165720 (+ 1964us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:41.165751 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:41.165892 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:41.177081 (+ 11189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:41.177089 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:41.177126 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:41.177542 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:41.177555 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.178873 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.178880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:41.181210 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:41.181226 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.183173 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.183183 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:41.183848 (+ 665us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:41.183882 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 08:44:41.183973 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 08:44:41.184121 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1803,"mutex_wait_us":30,"thread_start_us":624,"threads_started":1}
I20260928 08:44:41.184650 7744 tablet_service.cc:1467] Processing CreateTablet for tablet d3deecb68bf342b48ce6a312a11dcb11 (DEFAULT_TABLE table=test-table [id=b2f68156fd0340e69e0122c2352fe9ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:41.185096 7744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3deecb68bf342b48ce6a312a11dcb11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:41.185217 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.164293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57590 (local address 127.0.97.1:46847)
0928 08:44:41.176752 (+ 12459us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:41.176758 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:41.176779 (+ 21us) server_negotiation.cc:407] Connection header received
0928 08:44:41.176847 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:41.176852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:41.176927 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:41.177043 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:41.177755 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.178731 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.181414 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.182772 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.182897 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:41.182994 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:41.184741 (+ 1747us) server_negotiation.cc:299] Negotiation successful
0928 08:44:41.184919 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12278,"thread_start_us":12860,"threads_started":1}
I20260928 08:44:41.185877 7903 tablet_bootstrap.cc:654] T d3deecb68bf342b48ce6a312a11dcb11 P 1137082ae436457c84ba8100f189c4ec: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:41.185894 7455 tablet_service.cc:1467] Processing CreateTablet for tablet d3deecb68bf342b48ce6a312a11dcb11 (DEFAULT_TABLE table=test-table [id=b2f68156fd0340e69e0122c2352fe9ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:41.186349 7455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3deecb68bf342b48ce6a312a11dcb11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:41.188114 7903 tablet_bootstrap.cc:492] T d3deecb68bf342b48ce6a312a11dcb11 P 1137082ae436457c84ba8100f189c4ec: No bootstrap required, opened a new log
I20260928 08:44:41.188251 7903 ts_tablet_manager.cc:1397] T d3deecb68bf342b48ce6a312a11dcb11 P 1137082ae436457c84ba8100f189c4ec: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 08:44:41.189018 7903 raft_consensus.cc:348] T d3deecb68bf342b48ce6a312a11dcb11 P 1137082ae436457c84ba8100f189c4ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8863ce64e24035aaeed4bf282343b3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41011 } } peers { permanent_uuid: "1137082ae436457c84ba8100f189c4ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42561 } } peers { permanent_uuid: "fb125058c6c34aae8dff092e2f334d3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46847 } }
I20260928 08:44:41.189268 7903 raft_consensus.cc:374] T d3deecb68bf342b48ce6a312a11dcb11 P 1137082ae436457c84ba8100f189c4ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:41.189388 7903 raft_consensus.cc:729] T d3deecb68bf342b48ce6a312a11dcb11 P 1137082ae436457c84ba8100f189c4ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1137082ae436457c84ba8100f189c4ec, State: Initialized, Role: FOLLOWER
I20260928 08:44:41.189671 7903 consensus_queue.cc:260] T d3deecb68bf342b48ce6a312a11dcb11 P 1137082ae436457c84ba8100f189c4ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8863ce64e24035aaeed4bf282343b3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41011 } } peers { permanent_uuid: "1137082ae436457c84ba8100f189c4ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42561 } } peers { permanent_uuid: "fb125058c6c34aae8dff092e2f334d3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46847 } }
I20260928 08:44:41.192721 7903 ts_tablet_manager.cc:1428] T d3deecb68bf342b48ce6a312a11dcb11 P 1137082ae436457c84ba8100f189c4ec: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260928 08:44:41.193497 7669 heartbeater.cc:503] Master 127.0.97.60:37393 was elected leader, sending a full tablet report...
I20260928 08:44:41.194857 7907 tablet_bootstrap.cc:492] T d3deecb68bf342b48ce6a312a11dcb11 P be8863ce64e24035aaeed4bf282343b3: Bootstrap starting.
I20260928 08:44:41.196602 7907 tablet_bootstrap.cc:654] T d3deecb68bf342b48ce6a312a11dcb11 P be8863ce64e24035aaeed4bf282343b3: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:41.196691 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.190802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.97.2:52703)
0928 08:44:41.196247 (+ 5445us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:41.196403 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5313,"thread_start_us":5178,"threads_started":1}
I20260928 08:44:41.197418 7908 tablet_bootstrap.cc:492] T d3deecb68bf342b48ce6a312a11dcb11 P fb125058c6c34aae8dff092e2f334d3b: Bootstrap starting.
I20260928 08:44:41.197947 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.197385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.97.2:57901)
0928 08:44:41.197629 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:41.197764 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260928 08:44:41.199113 7907 tablet_bootstrap.cc:492] T d3deecb68bf342b48ce6a312a11dcb11 P be8863ce64e24035aaeed4bf282343b3: No bootstrap required, opened a new log
I20260928 08:44:41.199201 7908 tablet_bootstrap.cc:654] T d3deecb68bf342b48ce6a312a11dcb11 P fb125058c6c34aae8dff092e2f334d3b: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:41.199296 7907 ts_tablet_manager.cc:1397] T d3deecb68bf342b48ce6a312a11dcb11 P be8863ce64e24035aaeed4bf282343b3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 08:44:41.200103 7907 raft_consensus.cc:348] T d3deecb68bf342b48ce6a312a11dcb11 P be8863ce64e24035aaeed4bf282343b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8863ce64e24035aaeed4bf282343b3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41011 } } peers { permanent_uuid: "1137082ae436457c84ba8100f189c4ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42561 } } peers { permanent_uuid: "fb125058c6c34aae8dff092e2f334d3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46847 } }
I20260928 08:44:41.200327 7907 raft_consensus.cc:374] T d3deecb68bf342b48ce6a312a11dcb11 P be8863ce64e24035aaeed4bf282343b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:41.200441 7907 raft_consensus.cc:729] T d3deecb68bf342b48ce6a312a11dcb11 P be8863ce64e24035aaeed4bf282343b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be8863ce64e24035aaeed4bf282343b3, State: Initialized, Role: FOLLOWER
I20260928 08:44:41.200686 7907 consensus_queue.cc:260] T d3deecb68bf342b48ce6a312a11dcb11 P be8863ce64e24035aaeed4bf282343b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8863ce64e24035aaeed4bf282343b3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41011 } } peers { permanent_uuid: "1137082ae436457c84ba8100f189c4ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42561 } } peers { permanent_uuid: "fb125058c6c34aae8dff092e2f334d3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46847 } }
I20260928 08:44:41.201370 7908 tablet_bootstrap.cc:492] T d3deecb68bf342b48ce6a312a11dcb11 P fb125058c6c34aae8dff092e2f334d3b: No bootstrap required, opened a new log
I20260928 08:44:41.201546 7908 ts_tablet_manager.cc:1397] T d3deecb68bf342b48ce6a312a11dcb11 P fb125058c6c34aae8dff092e2f334d3b: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260928 08:44:41.202659 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.201699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.97.3:52419)
0928 08:44:41.202246 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:41.202426 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":301,"threads_started":1}
I20260928 08:44:41.202939 7908 raft_consensus.cc:348] T d3deecb68bf342b48ce6a312a11dcb11 P fb125058c6c34aae8dff092e2f334d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8863ce64e24035aaeed4bf282343b3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41011 } } peers { permanent_uuid: "1137082ae436457c84ba8100f189c4ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42561 } } peers { permanent_uuid: "fb125058c6c34aae8dff092e2f334d3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46847 } }
I20260928 08:44:41.203150 7907 ts_tablet_manager.cc:1428] T d3deecb68bf342b48ce6a312a11dcb11 P be8863ce64e24035aaeed4bf282343b3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 08:44:41.203179 7908 raft_consensus.cc:374] T d3deecb68bf342b48ce6a312a11dcb11 P fb125058c6c34aae8dff092e2f334d3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:41.203474 7908 raft_consensus.cc:729] T d3deecb68bf342b48ce6a312a11dcb11 P fb125058c6c34aae8dff092e2f334d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb125058c6c34aae8dff092e2f334d3b, State: Initialized, Role: FOLLOWER
I20260928 08:44:41.203585 7810 heartbeater.cc:503] Master 127.0.97.60:37393 was elected leader, sending a full tablet report...
I20260928 08:44:41.203728 7908 consensus_queue.cc:260] T d3deecb68bf342b48ce6a312a11dcb11 P fb125058c6c34aae8dff092e2f334d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8863ce64e24035aaeed4bf282343b3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41011 } } peers { permanent_uuid: "1137082ae436457c84ba8100f189c4ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42561 } } peers { permanent_uuid: "fb125058c6c34aae8dff092e2f334d3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46847 } }
I20260928 08:44:41.205289 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.203954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.97.3:55759)
0928 08:44:41.204983 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:41.205089 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":908}
I20260928 08:44:41.208071 7908 ts_tablet_manager.cc:1428] T d3deecb68bf342b48ce6a312a11dcb11 P fb125058c6c34aae8dff092e2f334d3b: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 08:44:41.209374 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.208349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.97.1:45163)
0928 08:44:41.208969 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:41.209124 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":428,"threads_started":1}
I20260928 08:44:41.210374 7520 heartbeater.cc:503] Master 127.0.97.60:37393 was elected leader, sending a full tablet report...
I20260928 08:44:41.210479 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.210105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.97.1:52437)
0928 08:44:41.210228 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:41.210328 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 08:44:41.445662 7911 raft_consensus.cc:482] T d3deecb68bf342b48ce6a312a11dcb11 P be8863ce64e24035aaeed4bf282343b3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:44:41.445899 7911 raft_consensus.cc:504] T d3deecb68bf342b48ce6a312a11dcb11 P be8863ce64e24035aaeed4bf282343b3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8863ce64e24035aaeed4bf282343b3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41011 } } peers { permanent_uuid: "1137082ae436457c84ba8100f189c4ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42561 } } peers { permanent_uuid: "fb125058c6c34aae8dff092e2f334d3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46847 } }
I20260928 08:44:41.447603 7911 leader_election.cc:288] T d3deecb68bf342b48ce6a312a11dcb11 P be8863ce64e24035aaeed4bf282343b3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1137082ae436457c84ba8100f189c4ec (127.0.97.2:42561), fb125058c6c34aae8dff092e2f334d3b (127.0.97.1:46847)
I20260928 08:44:41.453094 7912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.447878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:46847 (local address 127.0.97.3:33119)
0928 08:44:41.448109 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:41.448129 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:41.448292 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:41.448684 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:41.448688 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:41.448727 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:41.448982 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:41.448991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.449916 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.449924 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:41.451196 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:41.451209 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.452398 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.452404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:41.452564 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:41.452585 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 08:44:41.452687 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 08:44:41.452821 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260928 08:44:41.453078 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.448033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33119 (local address 127.0.97.1:46847)
0928 08:44:41.448248 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:41.448254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:41.448272 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:44:41.448506 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:41.448510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:41.448568 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:41.448663 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:41.449122 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.449786 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.451359 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.452214 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.452348 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:41.452452 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:41.452710 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 08:44:41.452826 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 08:44:41.453971 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.447940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47951 (local address 127.0.97.2:42561)
0928 08:44:41.448159 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:41.448166 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:41.448990 (+ 824us) server_negotiation.cc:407] Connection header received
0928 08:44:41.449206 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:41.449213 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:41.449283 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:41.449389 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:41.449899 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.450587 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.452331 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.453077 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.453230 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:41.453337 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:41.453598 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 08:44:41.453736 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 08:44:41.454070 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.447888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:42561 (local address 127.0.97.3:47951)
0928 08:44:41.448860 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:41.448883 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:41.449057 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:41.449402 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:41.449407 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:41.449430 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:41.449728 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:41.449737 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.450732 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.450740 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:41.452171 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:41.452190 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.453262 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.453268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:41.453436 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:41.453458 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 08:44:41.453575 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 08:44:41.453683 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":852,"mutex_wait_us":215,"thread_start_us":426,"threads_started":1}
I20260928 08:44:41.454166 7475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3deecb68bf342b48ce6a312a11dcb11" candidate_uuid: "be8863ce64e24035aaeed4bf282343b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb125058c6c34aae8dff092e2f334d3b" is_pre_election: true
I20260928 08:44:41.454605 7475 raft_consensus.cc:2454] T d3deecb68bf342b48ce6a312a11dcb11 P fb125058c6c34aae8dff092e2f334d3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be8863ce64e24035aaeed4bf282343b3 in term 0.
I20260928 08:44:41.454675 7624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3deecb68bf342b48ce6a312a11dcb11" candidate_uuid: "be8863ce64e24035aaeed4bf282343b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1137082ae436457c84ba8100f189c4ec" is_pre_election: true
I20260928 08:44:41.454969 7624 raft_consensus.cc:2454] T d3deecb68bf342b48ce6a312a11dcb11 P 1137082ae436457c84ba8100f189c4ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be8863ce64e24035aaeed4bf282343b3 in term 0.
I20260928 08:44:41.455556 7701 leader_election.cc:302] T d3deecb68bf342b48ce6a312a11dcb11 P be8863ce64e24035aaeed4bf282343b3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be8863ce64e24035aaeed4bf282343b3, fb125058c6c34aae8dff092e2f334d3b; no voters:
I20260928 08:44:41.456099 7911 raft_consensus.cc:2781] T d3deecb68bf342b48ce6a312a11dcb11 P be8863ce64e24035aaeed4bf282343b3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:44:41.456246 7911 raft_consensus.cc:482] T d3deecb68bf342b48ce6a312a11dcb11 P be8863ce64e24035aaeed4bf282343b3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:44:41.456382 7911 raft_consensus.cc:3037] T d3deecb68bf342b48ce6a312a11dcb11 P be8863ce64e24035aaeed4bf282343b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:41.460358 7911 raft_consensus.cc:504] T d3deecb68bf342b48ce6a312a11dcb11 P be8863ce64e24035aaeed4bf282343b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8863ce64e24035aaeed4bf282343b3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41011 } } peers { permanent_uuid: "1137082ae436457c84ba8100f189c4ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42561 } } peers { permanent_uuid: "fb125058c6c34aae8dff092e2f334d3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46847 } }
I20260928 08:44:41.461022 7911 leader_election.cc:288] T d3deecb68bf342b48ce6a312a11dcb11 P be8863ce64e24035aaeed4bf282343b3 [CANDIDATE]: Term 1 election: Requested vote from peers 1137082ae436457c84ba8100f189c4ec (127.0.97.2:42561), fb125058c6c34aae8dff092e2f334d3b (127.0.97.1:46847)
I20260928 08:44:41.461475 7624 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3deecb68bf342b48ce6a312a11dcb11" candidate_uuid: "be8863ce64e24035aaeed4bf282343b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1137082ae436457c84ba8100f189c4ec"
I20260928 08:44:41.461510 7475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3deecb68bf342b48ce6a312a11dcb11" candidate_uuid: "be8863ce64e24035aaeed4bf282343b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb125058c6c34aae8dff092e2f334d3b"
I20260928 08:44:41.461695 7624 raft_consensus.cc:3037] T d3deecb68bf342b48ce6a312a11dcb11 P 1137082ae436457c84ba8100f189c4ec [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:41.461707 7475 raft_consensus.cc:3037] T d3deecb68bf342b48ce6a312a11dcb11 P fb125058c6c34aae8dff092e2f334d3b [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:41.464789 7624 raft_consensus.cc:2454] T d3deecb68bf342b48ce6a312a11dcb11 P 1137082ae436457c84ba8100f189c4ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be8863ce64e24035aaeed4bf282343b3 in term 1.
I20260928 08:44:41.464797 7475 raft_consensus.cc:2454] T d3deecb68bf342b48ce6a312a11dcb11 P fb125058c6c34aae8dff092e2f334d3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be8863ce64e24035aaeed4bf282343b3 in term 1.
I20260928 08:44:41.465221 7699 leader_election.cc:302] T d3deecb68bf342b48ce6a312a11dcb11 P be8863ce64e24035aaeed4bf282343b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1137082ae436457c84ba8100f189c4ec, be8863ce64e24035aaeed4bf282343b3; no voters:
I20260928 08:44:41.465521 7911 raft_consensus.cc:2781] T d3deecb68bf342b48ce6a312a11dcb11 P be8863ce64e24035aaeed4bf282343b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:44:41.465922 7911 raft_consensus.cc:686] T d3deecb68bf342b48ce6a312a11dcb11 P be8863ce64e24035aaeed4bf282343b3 [term 1 LEADER]: Becoming Leader. State: Replica: be8863ce64e24035aaeed4bf282343b3, State: Running, Role: LEADER
I20260928 08:44:41.466356 7911 consensus_queue.cc:237] T d3deecb68bf342b48ce6a312a11dcb11 P be8863ce64e24035aaeed4bf282343b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8863ce64e24035aaeed4bf282343b3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41011 } } peers { permanent_uuid: "1137082ae436457c84ba8100f189c4ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42561 } } peers { permanent_uuid: "fb125058c6c34aae8dff092e2f334d3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46847 } }
I20260928 08:44:41.466944 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.466522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.97.3:38207)
0928 08:44:41.466664 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:41.466820 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 08:44:41.470006 7341 catalog_manager.cc:5495] T d3deecb68bf342b48ce6a312a11dcb11 P be8863ce64e24035aaeed4bf282343b3 reported cstate change: term changed from 0 to 1, leader changed from <none> to be8863ce64e24035aaeed4bf282343b3 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "be8863ce64e24035aaeed4bf282343b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be8863ce64e24035aaeed4bf282343b3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 41011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1137082ae436457c84ba8100f189c4ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb125058c6c34aae8dff092e2f334d3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46847 } health_report { overall_health: UNKNOWN } } }
I20260928 08:44:41.543905 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.539215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36842 (local address 127.0.97.3:41011)
0928 08:44:41.539417 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:41.539422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:41.539439 (+ 17us) server_negotiation.cc:407] Connection header received
0928 08:44:41.539575 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:41.539579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:41.539656 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:41.539772 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:44:41.540100 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.540767 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.541940 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.542323 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.542408 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:41.542742 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:44:41.543283 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:44:41.543386 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:41.543494 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 08:44:41.543553 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 08:44:41.548426 7624 raft_consensus.cc:1261] T d3deecb68bf342b48ce6a312a11dcb11 P 1137082ae436457c84ba8100f189c4ec [term 1 FOLLOWER]: Refusing update from remote peer be8863ce64e24035aaeed4bf282343b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:44:41.548943 7475 raft_consensus.cc:1261] T d3deecb68bf342b48ce6a312a11dcb11 P fb125058c6c34aae8dff092e2f334d3b [term 1 FOLLOWER]: Refusing update from remote peer be8863ce64e24035aaeed4bf282343b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:44:41.549124 7916 consensus_queue.cc:1035] T d3deecb68bf342b48ce6a312a11dcb11 P be8863ce64e24035aaeed4bf282343b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1137082ae436457c84ba8100f189c4ec" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 42561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:44:41.549646 7911 consensus_queue.cc:1035] T d3deecb68bf342b48ce6a312a11dcb11 P be8863ce64e24035aaeed4bf282343b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb125058c6c34aae8dff092e2f334d3b" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 46847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:44:41.552608 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.551987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.97.2:51831)
0928 08:44:41.552177 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:41.552339 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 08:44:41.552708 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.552295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.97.1:35429)
0928 08:44:41.552453 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:41.552534 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 08:44:41.560042 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.559329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.97.3:60551)
0928 08:44:41.559716 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:41.559864 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283}
I20260928 08:44:41.560710 7920 mvcc.cc:205] Tried to move back new op lower bound from 7334236494015086592 to 7334236493690454016. Current Snapshot: MvccSnapshot[applied={T|T < 7334236494015086592 or (T in {7334236494015086592})}]
I20260928 08:44:41.562872 7915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.562424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.97.3:54337)
0928 08:44:41.562617 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:41.562699 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260928 08:44:41.564411 7906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.563958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.97.2:40375)
0928 08:44:41.564157 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:41.564252 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260928 08:44:41.569516 7914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.559215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.97.1:50683)
0928 08:44:41.569182 (+ 9967us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:41.569300 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9819}
I20260928 08:44:41.571434 7341 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:44:41.571794 7341 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40216:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:44:41.578900 7341 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a63285bae4b64ecca10248bb821d965f: Sending DeleteTablet for 3 replicas of tablet d3deecb68bf342b48ce6a312a11dcb11
I20260928 08:44:41.579726 7744 tablet_service.cc:1514] Processing DeleteTablet for tablet d3deecb68bf342b48ce6a312a11dcb11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:41 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:36828
I20260928 08:44:41.580329 7604 tablet_service.cc:1514] Processing DeleteTablet for tablet d3deecb68bf342b48ce6a312a11dcb11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:41 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:53080
I20260928 08:44:41.580407 7929 tablet_replica.cc:330] T d3deecb68bf342b48ce6a312a11dcb11 P be8863ce64e24035aaeed4bf282343b3: stopping tablet replica
I20260928 08:44:41.580559 7929 raft_consensus.cc:2229] T d3deecb68bf342b48ce6a312a11dcb11 P be8863ce64e24035aaeed4bf282343b3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:44:41.580981 7455 tablet_service.cc:1514] Processing DeleteTablet for tablet d3deecb68bf342b48ce6a312a11dcb11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:41 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:57590
I20260928 08:44:41.581103 7929 raft_consensus.cc:2258] T d3deecb68bf342b48ce6a312a11dcb11 P be8863ce64e24035aaeed4bf282343b3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:41.581569 7931 tablet_replica.cc:330] T d3deecb68bf342b48ce6a312a11dcb11 P fb125058c6c34aae8dff092e2f334d3b: stopping tablet replica
I20260928 08:44:41.581763 7931 raft_consensus.cc:2229] T d3deecb68bf342b48ce6a312a11dcb11 P fb125058c6c34aae8dff092e2f334d3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:44:41.581727 7603 tablet_service.cc:1514] Processing DeleteTablet for tablet d3deecb68bf342b48ce6a312a11dcb11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:41 UTC) from {username='oryx', principal='oryx/127.0.97.60@KRBTEST.COM'} at 127.0.0.1:53080
I20260928 08:44:41.582068 7931 raft_consensus.cc:2258] T d3deecb68bf342b48ce6a312a11dcb11 P fb125058c6c34aae8dff092e2f334d3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:41.583129 7931 ts_tablet_manager.cc:1905] T d3deecb68bf342b48ce6a312a11dcb11 P fb125058c6c34aae8dff092e2f334d3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:41.583843 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 7328
I20260928 08:44:41.584425 7929 ts_tablet_manager.cc:1905] T d3deecb68bf342b48ce6a312a11dcb11 P be8863ce64e24035aaeed4bf282343b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:41.591626 7929 ts_tablet_manager.cc:1918] T d3deecb68bf342b48ce6a312a11dcb11 P be8863ce64e24035aaeed4bf282343b3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:44:41.591849 7930 tablet_replica.cc:330] T d3deecb68bf342b48ce6a312a11dcb11 P 1137082ae436457c84ba8100f189c4ec: stopping tablet replica
I20260928 08:44:41.592074 7930 raft_consensus.cc:2229] T d3deecb68bf342b48ce6a312a11dcb11 P 1137082ae436457c84ba8100f189c4ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:44:41.592208 7931 ts_tablet_manager.cc:1918] T d3deecb68bf342b48ce6a312a11dcb11 P fb125058c6c34aae8dff092e2f334d3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:44:41.592437 7930 raft_consensus.cc:2258] T d3deecb68bf342b48ce6a312a11dcb11 P 1137082ae436457c84ba8100f189c4ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:41.592458 7931 log.cc:1196] T d3deecb68bf342b48ce6a312a11dcb11 P fb125058c6c34aae8dff092e2f334d3b: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-0/wal/wals/d3deecb68bf342b48ce6a312a11dcb11
I20260928 08:44:41.593119 7931 ts_tablet_manager.cc:1939] T d3deecb68bf342b48ce6a312a11dcb11 P fb125058c6c34aae8dff092e2f334d3b: Deleting consensus metadata
I20260928 08:44:41.593472 7930 ts_tablet_manager.cc:1905] T d3deecb68bf342b48ce6a312a11dcb11 P 1137082ae436457c84ba8100f189c4ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:41.594636 7327 catalog_manager.cc:4841] TS fb125058c6c34aae8dff092e2f334d3b (127.0.97.1:46847): tablet d3deecb68bf342b48ce6a312a11dcb11 (table test-table [id=b2f68156fd0340e69e0122c2352fe9ae]) successfully deleted
I20260928 08:44:41.600425 7930 ts_tablet_manager.cc:1918] T d3deecb68bf342b48ce6a312a11dcb11 P 1137082ae436457c84ba8100f189c4ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:44:41.600600 7930 log.cc:1196] T d3deecb68bf342b48ce6a312a11dcb11 P 1137082ae436457c84ba8100f189c4ec: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-1/wal/wals/d3deecb68bf342b48ce6a312a11dcb11
I20260928 08:44:41.601143 7930 ts_tablet_manager.cc:1939] T d3deecb68bf342b48ce6a312a11dcb11 P 1137082ae436457c84ba8100f189c4ec: Deleting consensus metadata
I20260928 08:44:41.602335 7325 catalog_manager.cc:4841] TS 1137082ae436457c84ba8100f189c4ec (127.0.97.2:42561): tablet d3deecb68bf342b48ce6a312a11dcb11 (table test-table [id=b2f68156fd0340e69e0122c2352fe9ae]) successfully deleted
I20260928 08:44:41.602710 7929 log.cc:1196] T d3deecb68bf342b48ce6a312a11dcb11 P be8863ce64e24035aaeed4bf282343b3: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790585022442112-388-0/minicluster-data/ts-2/wal/wals/d3deecb68bf342b48ce6a312a11dcb11
W20260928 08:44:41.602901 7325 catalog_manager.cc:4808] TS 1137082ae436457c84ba8100f189c4ec (127.0.97.2:42561): delete failed for tablet d3deecb68bf342b48ce6a312a11dcb11 because the tablet was not found. No further retry: Not found: Tablet not found: d3deecb68bf342b48ce6a312a11dcb11
I20260928 08:44:41.603309 7929 ts_tablet_manager.cc:1939] T d3deecb68bf342b48ce6a312a11dcb11 P be8863ce64e24035aaeed4bf282343b3: Deleting consensus metadata
I20260928 08:44:41.604526 7326 catalog_manager.cc:4841] TS be8863ce64e24035aaeed4bf282343b3 (127.0.97.3:41011): tablet d3deecb68bf342b48ce6a312a11dcb11 (table test-table [id=b2f68156fd0340e69e0122c2352fe9ae]) successfully deleted
I20260928 08:44:41.613778 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.585027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57604 (local address 127.0.97.1:46847)
0928 08:44:41.585194 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:41.585201 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:41.595510 (+ 10309us) server_negotiation.cc:407] Connection header received
0928 08:44:41.595715 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:41.595720 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:41.595795 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:41.595933 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:41.597202 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.597923 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.599081 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.599461 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.599593 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:41.607617 (+ 8024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:41.607631 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:41.607669 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:41.607706 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:41.610997 (+ 3291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:41.611710 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:41.611723 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:41.611725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:41.611916 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:41.612268 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:41.612276 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:41.612279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:41.612630 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:41.612909 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:41.613231 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 08:44:41.613442 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 08:44:41.614506 7515 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:41.631937 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.631389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.0.1:36500)
0928 08:44:41.631647 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:41.631746 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260928 08:44:41.633198 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.632792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.0.1:36502)
0928 08:44:41.632965 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:41.633028 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260928 08:44:41.731082 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 7328
I20260928 08:44:41.753393 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 7532
I20260928 08:44:41.766412 7900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.754188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53088 (local address 127.0.97.2:42561)
0928 08:44:41.754351 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:41.754356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:41.754402 (+ 46us) server_negotiation.cc:407] Connection header received
0928 08:44:41.754605 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:41.754614 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:41.754675 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:41.754792 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:41.756111 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.756886 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.758080 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.758453 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.758596 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:41.761075 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:41.761099 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:41.761114 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:41.761144 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:41.763892 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:41.764526 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:41.764535 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:41.764539 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:41.764662 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:41.765021 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:41.765026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:41.765029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:41.765345 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:41.765621 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:41.765894 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 08:44:41.766106 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 08:44:41.767072 7664 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:41.858505 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 7532
I20260928 08:44:41.876340 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 7682
I20260928 08:44:41.891734 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:41.877236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36856 (local address 127.0.97.3:41011)
0928 08:44:41.877426 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:41.877432 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:41.877454 (+ 22us) server_negotiation.cc:407] Connection header received
0928 08:44:41.877518 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:41.877524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:41.877609 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:41.877738 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:41.878975 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.880392 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.881548 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:41.882267 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:41.882389 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:41.884817 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:41.884833 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:41.884855 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:41.884897 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:41.888967 (+ 4070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:41.889647 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:41.889665 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:41.889668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:41.889829 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:41.890147 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:41.890152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:41.890156 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:41.890602 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:41.890914 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:41.891110 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 08:44:41.891347 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 08:44:41.892621 7805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:42.012288 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:42.011792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.0.1:36512)
0928 08:44:42.012034 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:42.012151 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260928 08:44:42.013259 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:42.012935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46083 (local address 127.0.0.1:36516)
0928 08:44:42.013070 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:42.013142 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 08:44:42.019912 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 7682
I20260928 08:44:42.036989 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 7013
I20260928 08:44:42.038102 7180 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:42.153625 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 7013
I20260928 08:44:42.173916 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 7114
I20260928 08:44:42.174983 7371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:42.293708 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 7114
2026-09-28T08:44:42Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9949 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:44:42 dist-test-slave-n6xs krb5kdc[7940](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:44:42 dist-test-slave-n6xs krb5kdc[7940](info): set up 2 sockets
Sep 28 08:44:42 dist-test-slave-n6xs krb5kdc[7940](info): commencing operation
krb5kdc: starting...
W20260928 08:44:43.392748 388 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:44:43 dist-test-slave-n6xs krb5kdc[7940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:44:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:44:43Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/krb5kdc/oryx_127.0.97.62.keytab.
Entry for principal oryx/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/krb5kdc/oryx_127.0.97.62.keytab.
I20260928 08:44:43.603600 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46411
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:43.917207 7953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:43.917541 7953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:43.917721 7953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:43.924821 7953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:44:43.924968 7953 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 08:44:43.925079 7953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:43.925194 7953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:43.925287 7953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:44:43.925370 7953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:44:43.935242 7953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46411
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/krb5kdc/oryx_127.0.97.62.keytab
--principal=oryx/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:43.936584 7953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:43.937930 7953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:43.950003 7958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:43.950239 7959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:43.950634 7961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:43.950937 7953 server_base.cc:1034] running on GCE node
I20260928 08:44:43.951758 7953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:43.953145 7953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:43.954370 7953 hybrid_clock.cc:648] HybridClock initialized: now 1790585083954307 us; error 43 us; skew 500 ppm
I20260928 08:44:43.954648 7953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:43 dist-test-slave-n6xs krb5kdc[7940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585083, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:43.957692 7953 init.cc:377] Logged in from keytab as oryx/127.0.97.62@KRBTEST.COM (short username oryx)
I20260928 08:44:43.959157 7953 webserver.cc:460] Webserver started at http://127.0.97.62:34365/ using document root <none> and password file <none>
I20260928 08:44:43.959921 7953 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:43.960012 7953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:43.960376 7953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:43.963272 7953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "ef140c385b0b49229ff2e2ddd2a91894"
format_stamp: "Formatted at 2026-09-28 08:44:43 on dist-test-slave-n6xs"
server_key: "c82536bbd0a782d1c3ad9441f3bbf431"
server_key_iv: "085085971b42ed0245f26441e659ce3a"
server_key_version: "encryptionkey@0"
I20260928 08:44:43.964144 7953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "ef140c385b0b49229ff2e2ddd2a91894"
format_stamp: "Formatted at 2026-09-28 08:44:43 on dist-test-slave-n6xs"
server_key: "c82536bbd0a782d1c3ad9441f3bbf431"
server_key_iv: "085085971b42ed0245f26441e659ce3a"
server_key_version: "encryptionkey@0"
I20260928 08:44:43.969185 7953 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 08:44:43.972622 7968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:43.973959 7953 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:44:43.974171 7953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "ef140c385b0b49229ff2e2ddd2a91894"
format_stamp: "Formatted at 2026-09-28 08:44:43 on dist-test-slave-n6xs"
server_key: "c82536bbd0a782d1c3ad9441f3bbf431"
server_key_iv: "085085971b42ed0245f26441e659ce3a"
server_key_version: "encryptionkey@0"
I20260928 08:44:43.974385 7953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:43.986032 7953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:43.989785 7953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:43.990041 7953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:44.011154 7953 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46411
I20260928 08:44:44.011171 8029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46411 every 8 connection(s)
I20260928 08:44:44.012528 7953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:44:44.016367 8030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:44.022439 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 7953
I20260928 08:44:44.022680 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:44:44.023538 388 external_mini_cluster.cc:1428] Setting key e20f1c91fa8da8fbe987be6bd991de1b
I20260928 08:44:44.027938 8030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef140c385b0b49229ff2e2ddd2a91894: Bootstrap starting.
I20260928 08:44:44.031951 8030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef140c385b0b49229ff2e2ddd2a91894: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:44.033046 8030 log.cc:824] T 00000000000000000000000000000000 P ef140c385b0b49229ff2e2ddd2a91894: Log is configured to *not* fsync() on all Append() calls
I20260928 08:44:44.035923 8030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef140c385b0b49229ff2e2ddd2a91894: No bootstrap required, opened a new log
Sep 28 08:44:44 dist-test-slave-n6xs krb5kdc[7940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 08:44:44.045696 8030 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef140c385b0b49229ff2e2ddd2a91894 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef140c385b0b49229ff2e2ddd2a91894" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46411 } }
I20260928 08:44:44.046058 8030 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef140c385b0b49229ff2e2ddd2a91894 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:44.046172 8030 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef140c385b0b49229ff2e2ddd2a91894 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef140c385b0b49229ff2e2ddd2a91894, State: Initialized, Role: FOLLOWER
I20260928 08:44:44.046845 8030 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef140c385b0b49229ff2e2ddd2a91894 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef140c385b0b49229ff2e2ddd2a91894" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46411 } }
I20260928 08:44:44.047142 8030 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef140c385b0b49229ff2e2ddd2a91894 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:44:44.047298 8030 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef140c385b0b49229ff2e2ddd2a91894 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:44:44.047459 8030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef140c385b0b49229ff2e2ddd2a91894 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:44.050086 8033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:44.026310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35508 (local address 127.0.97.62:46411)
0928 08:44:44.027768 (+ 1458us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:44.027784 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:44.028325 (+ 541us) server_negotiation.cc:407] Connection header received
0928 08:44:44.030785 (+ 2460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:44.030813 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:44.031300 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:44.031731 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:44.033072 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:44.034967 (+ 1895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:44.036337 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:44.037374 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:44.037548 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:44.041210 (+ 3662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:44.041221 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:44.041240 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:44.041287 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:44.045489 (+ 4202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:44.046357 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:44.046364 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:44.046371 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:44.046596 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:44.046948 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:44.046953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:44.046957 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:44.047650 (+ 693us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:44.048035 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:44.048361 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 08:44:44.048778 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":1177,"thread_start_us":878,"threads_started":1}
I20260928 08:44:44.050658 8030 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef140c385b0b49229ff2e2ddd2a91894 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef140c385b0b49229ff2e2ddd2a91894" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46411 } }
I20260928 08:44:44.051196 8030 leader_election.cc:302] T 00000000000000000000000000000000 P ef140c385b0b49229ff2e2ddd2a91894 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef140c385b0b49229ff2e2ddd2a91894; no voters:
I20260928 08:44:44.051864 8030 leader_election.cc:288] T 00000000000000000000000000000000 P ef140c385b0b49229ff2e2ddd2a91894 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:44:44.051983 8035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef140c385b0b49229ff2e2ddd2a91894 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:44:44.053807 8035 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef140c385b0b49229ff2e2ddd2a91894 [term 1 LEADER]: Becoming Leader. State: Replica: ef140c385b0b49229ff2e2ddd2a91894, State: Running, Role: LEADER
I20260928 08:44:44.053889 8030 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef140c385b0b49229ff2e2ddd2a91894 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:44:44.054244 8035 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef140c385b0b49229ff2e2ddd2a91894 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef140c385b0b49229ff2e2ddd2a91894" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46411 } }
I20260928 08:44:44.057510 8037 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef140c385b0b49229ff2e2ddd2a91894 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef140c385b0b49229ff2e2ddd2a91894. Latest consensus state: current_term: 1 leader_uuid: "ef140c385b0b49229ff2e2ddd2a91894" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef140c385b0b49229ff2e2ddd2a91894" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46411 } } }
I20260928 08:44:44.057905 8037 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef140c385b0b49229ff2e2ddd2a91894 [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:44.057519 8036 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef140c385b0b49229ff2e2ddd2a91894 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef140c385b0b49229ff2e2ddd2a91894" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef140c385b0b49229ff2e2ddd2a91894" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46411 } } }
I20260928 08:44:44.058374 8036 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef140c385b0b49229ff2e2ddd2a91894 [sys.catalog]: This master's current role is: LEADER
W20260928 08:44:44.066830 8051 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ef140c385b0b49229ff2e2ddd2a91894: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:44:44.066956 8051 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:44:44.067086 8044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:44:44.069173 8044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:44:44.077495 8044 catalog_manager.cc:1347] Generated new cluster ID: 6150ccd61db54925904dc2ef1c98adf9
I20260928 08:44:44.077592 8044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:44:44.104188 8044 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:44:44.106096 8044 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:44:44.120324 8044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef140c385b0b49229ff2e2ddd2a91894: Generated new TSK 0
I20260928 08:44:44.121351 8044 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/krb5kdc/oryx_127.0.97.1.keytab.
I20260928 08:44:44.179595 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46411
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:44.500384 8056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:44.500653 8056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:44.500748 8056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:44.507782 8056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:44.507947 8056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:44.508056 8056 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:44:44.508140 8056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:44:44.517652 8056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:44.518939 8056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:44.520264 8056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:44.533814 8064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:44.533875 8061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:44.533852 8062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:44.534722 8056 server_base.cc:1034] running on GCE node
I20260928 08:44:44.535331 8056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:44.536370 8056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:44.537613 8056 hybrid_clock.cc:648] HybridClock initialized: now 1790585084537549 us; error 41 us; skew 500 ppm
I20260928 08:44:44.537919 8056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:44 dist-test-slave-n6xs krb5kdc[7940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585084, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:44.541107 8056 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260928 08:44:44.542498 8056 webserver.cc:460] Webserver started at http://127.0.97.1:41175/ using document root <none> and password file <none>
I20260928 08:44:44.543184 8056 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:44.543313 8056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:44.543628 8056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:44.546583 8056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-0/data/instance:
uuid: "e4860201f08e45c98eea6befb1984108"
format_stamp: "Formatted at 2026-09-28 08:44:44 on dist-test-slave-n6xs"
server_key: "d2054e9a1f230dd382640188d26d1a95"
server_key_iv: "473668b103c61f5c15ca89f7fc59f925"
server_key_version: "encryptionkey@0"
I20260928 08:44:44.547391 8056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-0/wal/instance:
uuid: "e4860201f08e45c98eea6befb1984108"
format_stamp: "Formatted at 2026-09-28 08:44:44 on dist-test-slave-n6xs"
server_key: "d2054e9a1f230dd382640188d26d1a95"
server_key_iv: "473668b103c61f5c15ca89f7fc59f925"
server_key_version: "encryptionkey@0"
I20260928 08:44:44.552521 8056 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 08:44:44.556102 8071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:44.557590 8056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 08:44:44.557812 8056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-0/wal
uuid: "e4860201f08e45c98eea6befb1984108"
format_stamp: "Formatted at 2026-09-28 08:44:44 on dist-test-slave-n6xs"
server_key: "d2054e9a1f230dd382640188d26d1a95"
server_key_iv: "473668b103c61f5c15ca89f7fc59f925"
server_key_version: "encryptionkey@0"
I20260928 08:44:44.558043 8056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:44.591377 8056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:44.595993 8056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:44.596304 8056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:44.598984 8056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:44:44.599109 8056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:44.599243 8056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:44:44.599313 8056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:44:44 dist-test-slave-n6xs krb5kdc[7940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585084, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 08:44:44.622525 8080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:44.599847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59663 (local address 127.0.97.62:46411)
0928 08:44:44.600341 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:44.600346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:44.601381 (+ 1035us) server_negotiation.cc:407] Connection header received
0928 08:44:44.603368 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:44.603374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:44.603495 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:44.603614 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:44.605891 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:44.607123 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:44.609455 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:44.609854 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:44.609982 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:44.614878 (+ 4896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:44.614887 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:44.614892 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:44.614941 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:44.618480 (+ 3539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:44.619536 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:44.619544 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:44.619550 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:44.619713 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:44.620485 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:44.620493 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:44.620502 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:44.620864 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:44.621032 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:44.621926 (+ 894us) server_negotiation.cc:299] Negotiation successful
0928 08:44:44.622143 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":287,"threads_started":1}
I20260928 08:44:44.623857 8081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:44.600433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46411 (local address 127.0.97.1:59663)
0928 08:44:44.601224 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:44.601263 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:44.603112 (+ 1849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:44.603948 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:44.603962 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:44.604778 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:44.605604 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:44.605632 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:44.607324 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:44.607331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:44.609298 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:44.609309 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:44.610099 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:44.610105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:44.610318 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:44.611194 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:44.611229 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:44.614552 (+ 3323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:44.618673 (+ 4121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:44.618684 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:44.618706 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:44.619329 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:44.619927 (+ 598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:44.619936 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:44.619943 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:44.620318 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:44.621005 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:44.621013 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:44.621526 (+ 513us) client_negotiation.cc:769] Sending connection context
0928 08:44:44.621843 (+ 317us) client_negotiation.cc:240] Negotiation successful
0928 08:44:44.622416 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":314,"threads_started":1}
I20260928 08:44:44.645919 8159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:44.630686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46411 (local address 127.0.97.1:37735)
0928 08:44:44.631713 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:44.631729 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:44.631892 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:44.632276 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:44.632279 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:44.632646 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:44.632910 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:44.632918 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:44.634704 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:44.634708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:44.635700 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:44.635713 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:44.636992 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:44.636997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:44.637119 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:44.637837 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:44.637865 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:44.639255 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:44.642726 (+ 3471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:44.642732 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:44.642736 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:44.643344 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:44.643770 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:44.643777 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:44.643783 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:44.643968 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:44.644656 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:44.644660 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:44.644855 (+ 195us) client_negotiation.cc:769] Sending connection context
0928 08:44:44.645023 (+ 168us) client_negotiation.cc:240] Negotiation successful
0928 08:44:44.645364 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"thread_start_us":732,"threads_started":1}
I20260928 08:44:44.646607 8080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:44.630791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37735 (local address 127.0.97.62:46411)
0928 08:44:44.630991 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:44.630997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:44.631832 (+ 835us) server_negotiation.cc:407] Connection header received
0928 08:44:44.632013 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:44.632022 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:44.632122 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:44.632261 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:44.633037 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:44.634519 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:44.635884 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:44.636779 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:44.636948 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:44.639485 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:44.639492 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:44.639497 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:44.639534 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:44.642588 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:44.643458 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:44.643465 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:44.643469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:44.643583 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:44.644107 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:44.644113 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:44.644116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:44.644485 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:44.644712 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:44.646220 (+ 1508us) server_negotiation.cc:299] Negotiation successful
0928 08:44:44.646388 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 08:44:44.650455 8056 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40955
I20260928 08:44:44.650535 8191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40955 every 8 connection(s)
I20260928 08:44:44.651639 8056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:44:44.653537 8192 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46411
I20260928 08:44:44.653798 8192 heartbeater.cc:464] Registering TS with master...
I20260928 08:44:44.654448 8192 heartbeater.cc:511] Master 127.0.97.62:46411 requested a full tablet report, sending...
I20260928 08:44:44.656322 7985 ts_manager.cc:194] Registered new tserver with Master: e4860201f08e45c98eea6befb1984108 (127.0.97.1:40955)
I20260928 08:44:44.657655 7985 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:59663
I20260928 08:44:44.659960 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 8056
I20260928 08:44:44.660183 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:44:44.660763 388 external_mini_cluster.cc:1428] Setting key f82f64b0350927f9a84e2ba2f84730bf
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/krb5kdc/oryx_127.0.97.2.keytab.
I20260928 08:44:44.704941 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46411
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:45.033546 8196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:45.033857 8196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:45.034037 8196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:45.041294 8196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:45.041432 8196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:45.041590 8196 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:44:45.041735 8196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:44:45.051457 8196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:45.052874 8196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:45.054263 8196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:45.067557 8201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:45.067567 8202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:45.067763 8204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:45.068464 8196 server_base.cc:1034] running on GCE node
I20260928 08:44:45.069083 8196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:45.069939 8196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:45.071189 8196 hybrid_clock.cc:648] HybridClock initialized: now 1790585085071121 us; error 27 us; skew 500 ppm
I20260928 08:44:45.071482 8196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:45 dist-test-slave-n6xs krb5kdc[7940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585085, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:45.074545 8196 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260928 08:44:45.075944 8196 webserver.cc:460] Webserver started at http://127.0.97.2:46847/ using document root <none> and password file <none>
I20260928 08:44:45.076545 8196 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:45.076622 8196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:45.076872 8196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:45.079636 8196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-1/data/instance:
uuid: "479cdfbbfa0a496abd505220830d412e"
format_stamp: "Formatted at 2026-09-28 08:44:45 on dist-test-slave-n6xs"
server_key: "a7890d23c1a84bd0a9406d512a389040"
server_key_iv: "7aec7d9e0b7e3af43c42dcc439e71534"
server_key_version: "encryptionkey@0"
I20260928 08:44:45.080439 8196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-1/wal/instance:
uuid: "479cdfbbfa0a496abd505220830d412e"
format_stamp: "Formatted at 2026-09-28 08:44:45 on dist-test-slave-n6xs"
server_key: "a7890d23c1a84bd0a9406d512a389040"
server_key_iv: "7aec7d9e0b7e3af43c42dcc439e71534"
server_key_version: "encryptionkey@0"
I20260928 08:44:45.085013 8196 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 08:44:45.088212 8211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:45.089471 8196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:44:45.089594 8196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-1/wal
uuid: "479cdfbbfa0a496abd505220830d412e"
format_stamp: "Formatted at 2026-09-28 08:44:45 on dist-test-slave-n6xs"
server_key: "a7890d23c1a84bd0a9406d512a389040"
server_key_iv: "7aec7d9e0b7e3af43c42dcc439e71534"
server_key_version: "encryptionkey@0"
I20260928 08:44:45.089756 8196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:45.111410 8196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:45.115267 8196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:45.115513 8196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:45.117940 8196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:44:45.118050 8196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:45.118189 8196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:44:45.118257 8196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:44:45 dist-test-slave-n6xs krb5kdc[7940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585085, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 08:44:45.144510 8220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:45.119437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46411 (local address 127.0.97.2:59789)
0928 08:44:45.120271 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:45.120330 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:45.122265 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:45.123077 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:45.123091 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:45.124038 (+ 947us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:45.124890 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:45.124917 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.126361 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.126369 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:45.129191 (+ 2822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:45.129206 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.130241 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.130253 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:45.130591 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:45.131391 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:45.131435 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:45.135402 (+ 3967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:45.139150 (+ 3748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:45.139162 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:45.139179 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:45.139931 (+ 752us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:45.140422 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:45.140432 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:45.140438 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:45.140761 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:45.141453 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:45.141462 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:45.141972 (+ 510us) client_negotiation.cc:769] Sending connection context
0928 08:44:45.142302 (+ 330us) client_negotiation.cc:240] Negotiation successful
0928 08:44:45.142893 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":396,"threads_started":1}
I20260928 08:44:45.145416 8080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:45.118778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:59789 (local address 127.0.97.62:46411)
0928 08:44:45.118965 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:45.118970 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:45.120452 (+ 1482us) server_negotiation.cc:407] Connection header received
0928 08:44:45.122585 (+ 2133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:45.122591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:45.122687 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:45.122808 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:45.125068 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.126182 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.129383 (+ 3201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.129935 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.130079 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:45.135696 (+ 5617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:45.135701 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:45.135705 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:45.135737 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:45.139013 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:45.140080 (+ 1067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:45.140088 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:45.140093 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:45.140294 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:45.140901 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:45.140906 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:45.140910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:45.141327 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:45.141478 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:45.145025 (+ 3547us) server_negotiation.cc:299] Negotiation successful
0928 08:44:45.145201 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 08:44:45.165705 8080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:45.150461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:57347 (local address 127.0.97.62:46411)
0928 08:44:45.150754 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:45.150763 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:45.152046 (+ 1283us) server_negotiation.cc:407] Connection header received
0928 08:44:45.152107 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:45.152110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:45.152165 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:45.152259 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:45.153723 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.154704 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.156895 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.157293 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.157416 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:45.159975 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:45.160001 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:45.160004 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:45.160053 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:45.162739 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:45.163700 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:45.163704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:45.163706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:45.163875 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:45.164241 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:45.164246 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:45.164249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:45.164572 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:45.164742 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:45.165121 (+ 379us) server_negotiation.cc:299] Negotiation successful
0928 08:44:45.165331 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 08:44:45.165853 8295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:45.150432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46411 (local address 127.0.97.2:57347)
0928 08:44:45.151648 (+ 1216us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:45.151673 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:45.151873 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:45.152326 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:45.152333 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:45.153003 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:45.153537 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:45.153558 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.154875 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.154882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:45.156744 (+ 1862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:45.156755 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.157484 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.157490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:45.157639 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:45.158312 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:45.158335 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:45.159699 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:45.162855 (+ 3156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:45.162863 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:45.162869 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:45.163522 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:45.163985 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:45.163992 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:45.163996 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:45.164132 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:45.164698 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:45.164703 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:45.164880 (+ 177us) client_negotiation.cc:769] Sending connection context
0928 08:44:45.165024 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 08:44:45.165343 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":998,"thread_start_us":833,"threads_started":1}
I20260928 08:44:45.174561 8196 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40415
I20260928 08:44:45.174602 8330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40415 every 8 connection(s)
I20260928 08:44:45.175787 8196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:44:45.177980 8331 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46411
I20260928 08:44:45.178177 8331 heartbeater.cc:464] Registering TS with master...
I20260928 08:44:45.178817 8331 heartbeater.cc:511] Master 127.0.97.62:46411 requested a full tablet report, sending...
I20260928 08:44:45.180091 7985 ts_manager.cc:194] Registered new tserver with Master: 479cdfbbfa0a496abd505220830d412e (127.0.97.2:40415)
I20260928 08:44:45.180896 7985 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:59789
I20260928 08:44:45.184470 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 8196
I20260928 08:44:45.184653 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:44:45.185073 388 external_mini_cluster.cc:1428] Setting key 8da32709eb8261fa836a477b0012ba6a
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260928 08:44:45.231187 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46411
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:45.562665 8335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:45.562999 8335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:45.563184 8335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:45.570391 8335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:45.570550 8335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:45.570776 8335 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 08:44:45.570950 8335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:44:45.581442 8335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.97.62:46411
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:45.582710 8335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:45.584156 8335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:45.597638 8340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:45.597648 8343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:45.597640 8341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:45.598927 8335 server_base.cc:1034] running on GCE node
I20260928 08:44:45.599529 8335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:45.600528 8335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:45.601775 8335 hybrid_clock.cc:648] HybridClock initialized: now 1790585085601695 us; error 60 us; skew 500 ppm
I20260928 08:44:45.602075 8335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:45 dist-test-slave-n6xs krb5kdc[7940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585085, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:45.605322 8335 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260928 08:44:45.606734 8335 webserver.cc:460] Webserver started at http://127.0.97.3:37625/ using document root <none> and password file <none>
I20260928 08:44:45.607436 8335 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:45.607565 8335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:45.607942 8335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:45.610759 8335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-2/data/instance:
uuid: "6951c64acb48467bb84a75d1b715b8c1"
format_stamp: "Formatted at 2026-09-28 08:44:45 on dist-test-slave-n6xs"
server_key: "c1b74e14e4f43f398d43b7c39f17c00d"
server_key_iv: "0f0b24844d1c076b62113b0490c0350c"
server_key_version: "encryptionkey@0"
I20260928 08:44:45.611548 8335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-2/wal/instance:
uuid: "6951c64acb48467bb84a75d1b715b8c1"
format_stamp: "Formatted at 2026-09-28 08:44:45 on dist-test-slave-n6xs"
server_key: "c1b74e14e4f43f398d43b7c39f17c00d"
server_key_iv: "0f0b24844d1c076b62113b0490c0350c"
server_key_version: "encryptionkey@0"
I20260928 08:44:45.616567 8335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260928 08:44:45.619917 8350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:45.621212 8335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:44:45.621378 8335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-2/wal
uuid: "6951c64acb48467bb84a75d1b715b8c1"
format_stamp: "Formatted at 2026-09-28 08:44:45 on dist-test-slave-n6xs"
server_key: "c1b74e14e4f43f398d43b7c39f17c00d"
server_key_iv: "0f0b24844d1c076b62113b0490c0350c"
server_key_version: "encryptionkey@0"
I20260928 08:44:45.621584 8335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:45.637784 8335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:45.642004 8335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:45.642274 8335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:45.644946 8335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:44:45.645092 8335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 08:44:45.645265 8335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:44:45.645346 8335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:45.660431 8192 heartbeater.cc:503] Master 127.0.97.62:46411 was elected leader, sending a full tablet report...
Sep 28 08:44:45 dist-test-slave-n6xs krb5kdc[7940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585085, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 08:44:45.670099 8080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:45.645510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33599 (local address 127.0.97.62:46411)
0928 08:44:45.645743 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:45.645749 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:45.647140 (+ 1391us) server_negotiation.cc:407] Connection header received
0928 08:44:45.649240 (+ 2100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:45.649250 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:45.649355 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:45.649461 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:45.652274 (+ 2813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.653423 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.656884 (+ 3461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.657539 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.657694 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:45.663030 (+ 5336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:45.663040 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:45.663047 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:45.663099 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:45.666641 (+ 3542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:45.667741 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:45.667745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:45.667747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:45.667925 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:45.668384 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:45.668389 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:45.668392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:45.668666 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:45.668824 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:45.669508 (+ 684us) server_negotiation.cc:299] Negotiation successful
0928 08:44:45.669762 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 08:44:45.671277 8359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:45.646096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46411 (local address 127.0.97.3:33599)
0928 08:44:45.646967 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:45.647018 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:45.648956 (+ 1938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:45.649971 (+ 1015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:45.649986 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:45.650999 (+ 1013us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:45.652067 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:45.652100 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.653617 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.653625 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:45.656669 (+ 3044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:45.656688 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.658204 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.658214 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:45.658567 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:45.659484 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:45.659504 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:45.662496 (+ 2992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:45.666817 (+ 4321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:45.666830 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:45.666848 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:45.667584 (+ 736us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:45.668038 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:45.668044 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:45.668047 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:45.668260 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:45.668790 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:45.668797 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:45.669133 (+ 336us) client_negotiation.cc:769] Sending connection context
0928 08:44:45.669396 (+ 263us) client_negotiation.cc:240] Negotiation successful
0928 08:44:45.669879 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":378,"threads_started":1}
I20260928 08:44:45.689703 8080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:45.676521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:52057 (local address 127.0.97.62:46411)
0928 08:44:45.676694 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:45.676698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:45.677781 (+ 1083us) server_negotiation.cc:407] Connection header received
0928 08:44:45.677987 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:45.677992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:45.678057 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:45.678162 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:45.679166 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.679907 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.681934 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.682367 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.682500 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:45.684207 (+ 1707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:45.684213 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:45.684217 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:45.684263 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:45.686769 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:45.687754 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:45.687762 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:45.687767 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:45.687967 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:45.688383 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:45.688390 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:45.688394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:45.688652 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:45.688795 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:45.689184 (+ 389us) server_negotiation.cc:299] Negotiation successful
0928 08:44:45.689382 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 08:44:45.689797 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:45.676659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46411 (local address 127.0.97.3:52057)
0928 08:44:45.677673 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:45.677696 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:45.677846 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:45.678179 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:45.678183 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:45.678714 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:45.679033 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:45.679042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.680031 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.680037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:45.681793 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:45.681806 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.682537 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.682543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:45.682725 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:45.683288 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:45.683302 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:45.684029 (+ 727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:45.686887 (+ 2858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:45.686893 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:45.686904 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:45.687568 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:45.688108 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:45.688114 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:45.688130 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:45.688259 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:45.688763 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:45.688767 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:45.688934 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 08:44:45.689078 (+ 144us) client_negotiation.cc:240] Negotiation successful
0928 08:44:45.689391 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"thread_start_us":629,"threads_started":1}
I20260928 08:44:45.702493 8335 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34497
I20260928 08:44:45.702543 8469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34497 every 8 connection(s)
I20260928 08:44:45.703665 8335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:44:45.705691 8470 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46411
I20260928 08:44:45.705915 8470 heartbeater.cc:464] Registering TS with master...
I20260928 08:44:45.706522 8470 heartbeater.cc:511] Master 127.0.97.62:46411 requested a full tablet report, sending...
I20260928 08:44:45.707773 7985 ts_manager.cc:194] Registered new tserver with Master: 6951c64acb48467bb84a75d1b715b8c1 (127.0.97.3:34497)
I20260928 08:44:45.708617 7985 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:33599
I20260928 08:44:45.712157 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 8335
I20260928 08:44:45.712304 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:44:45.712663 388 external_mini_cluster.cc:1428] Setting key eb9d643ecede1513a7699de9b53dea27
I20260928 08:44:45.715443 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 08:44:45 dist-test-slave-n6xs krb5kdc[7940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585085, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 08:44:45 dist-test-slave-n6xs krb5kdc[7940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:44:45 dist-test-slave-n6xs krb5kdc[7940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 08:44:45.764328 8080 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:44:45.752890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35516 (local address 127.0.97.62:46411)
0928 08:44:45.753122 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:45.753129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:45.753333 (+ 204us) server_negotiation.cc:407] Connection header received
0928 08:44:45.753542 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:45.753553 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:45.753615 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:45.753786 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:45.755079 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.755900 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.757217 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.757643 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.757781 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:45.764030 (+ 6249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35516: BlockingRecv error: recv got EOF from 127.0.0.1:35516 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 28 08:44:45 dist-test-slave-n6xs krb5kdc[7940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585085, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 08:44:45.780441 8080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:45.767290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35532 (local address 127.0.97.62:46411)
0928 08:44:45.767449 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:45.767453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:45.767760 (+ 307us) server_negotiation.cc:407] Connection header received
0928 08:44:45.767939 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:45.767944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:45.768024 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:45.768124 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:45.769188 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.769999 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.771181 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.771675 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.771786 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:45.775438 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:45.775444 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:45.775448 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:45.775480 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:45.778307 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:45.778868 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:45.778872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:45.778875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:45.778991 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:45.779320 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:45.779323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:45.779325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:45.779564 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:45.779857 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:45.780062 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 08:44:45.780222 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 08:44:45.784332 7985 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:44:45.791007 7985 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:44:45.813725 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:45.803195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46096 (local address 127.0.97.1:40955)
0928 08:44:45.805948 (+ 2753us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:45.805960 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:45.805994 (+ 34us) server_negotiation.cc:407] Connection header received
0928 08:44:45.806104 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:45.806115 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:45.806583 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:45.806770 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:45.807686 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.809068 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.811572 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.812550 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.812702 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:45.812900 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:45.813228 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 08:44:45.813393 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2551,"thread_start_us":2375,"threads_started":1}
I20260928 08:44:45.813804 8489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:45.802996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40955 (local address 127.0.0.1:46096)
0928 08:44:45.804970 (+ 1974us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:45.805068 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:45.805263 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:45.807028 (+ 1765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:45.807038 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:45.807082 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:45.807501 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:45.807511 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.809252 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.809259 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:45.811435 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:45.811448 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.812761 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.812767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:45.812961 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:45.812994 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 08:44:45.813159 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 08:44:45.813367 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1653,"mutex_wait_us":6,"thread_start_us":3368,"threads_started":1}
I20260928 08:44:45.815737 8488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:45.803206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34497 (local address 127.0.0.1:37942)
0928 08:44:45.806656 (+ 3450us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:45.806674 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:45.806792 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:45.807813 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:45.807841 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:45.807869 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:45.808419 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:45.808435 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.810445 (+ 2010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.810450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:45.812941 (+ 2491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:45.812955 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.814997 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.815003 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:45.815180 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:45.815215 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 08:44:45.815289 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 08:44:45.815429 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3294,"thread_start_us":1320,"threads_started":1}
I20260928 08:44:45.816329 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:45.803937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37942 (local address 127.0.97.3:34497)
0928 08:44:45.806374 (+ 2437us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:45.806385 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:45.806932 (+ 547us) server_negotiation.cc:407] Connection header received
0928 08:44:45.807051 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:45.807058 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:45.807332 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:45.807509 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:45.808645 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.810266 (+ 1621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.813116 (+ 2850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.814558 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.814675 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:45.814879 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:45.815776 (+ 897us) server_negotiation.cc:299] Negotiation successful
0928 08:44:45.815980 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2255,"thread_start_us":2071,"threads_started":1}
I20260928 08:44:45.816946 8492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:45.806937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53268 (local address 127.0.97.2:40415)
0928 08:44:45.807766 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:45.807774 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:45.808521 (+ 747us) server_negotiation.cc:407] Connection header received
0928 08:44:45.808702 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:45.808709 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:45.808891 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:45.809054 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:45.809691 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.811115 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.814472 (+ 3357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.815835 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.815963 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:45.816149 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:45.816633 (+ 484us) server_negotiation.cc:299] Negotiation successful
0928 08:44:45.816751 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":531,"threads_started":1}
I20260928 08:44:45.817668 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:45.807744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40415 (local address 127.0.0.1:53268)
0928 08:44:45.808395 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:45.808417 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:45.808549 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:45.809076 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:45.809081 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:45.809113 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:45.809509 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:45.809523 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.811278 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.811290 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:45.813641 (+ 2351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:45.813670 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.816282 (+ 2612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.816287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:45.816421 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:45.816466 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 08:44:45.817200 (+ 734us) client_negotiation.cc:240] Negotiation successful
0928 08:44:45.817339 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":328,"threads_started":1}
I20260928 08:44:45.818046 8122 tablet_service.cc:1467] Processing CreateTablet for tablet 4d82c36f50f045aea0835d931324b9f8 (DEFAULT_TABLE table=test-table [id=2b56e1c15cf349e6bdd62403f03aa494]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:45.819514 8122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d82c36f50f045aea0835d931324b9f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:45.819438 8400 tablet_service.cc:1467] Processing CreateTablet for tablet 4d82c36f50f045aea0835d931324b9f8 (DEFAULT_TABLE table=test-table [id=2b56e1c15cf349e6bdd62403f03aa494]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:45.820782 8400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d82c36f50f045aea0835d931324b9f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:45.821075 8261 tablet_service.cc:1467] Processing CreateTablet for tablet 4d82c36f50f045aea0835d931324b9f8 (DEFAULT_TABLE table=test-table [id=2b56e1c15cf349e6bdd62403f03aa494]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:45.822389 8261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d82c36f50f045aea0835d931324b9f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:45.831343 8494 tablet_bootstrap.cc:492] T 4d82c36f50f045aea0835d931324b9f8 P 6951c64acb48467bb84a75d1b715b8c1: Bootstrap starting.
I20260928 08:44:45.835110 8495 tablet_bootstrap.cc:492] T 4d82c36f50f045aea0835d931324b9f8 P e4860201f08e45c98eea6befb1984108: Bootstrap starting.
I20260928 08:44:45.835192 8494 tablet_bootstrap.cc:654] T 4d82c36f50f045aea0835d931324b9f8 P 6951c64acb48467bb84a75d1b715b8c1: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:45.836524 8494 log.cc:824] T 4d82c36f50f045aea0835d931324b9f8 P 6951c64acb48467bb84a75d1b715b8c1: Log is configured to *not* fsync() on all Append() calls
I20260928 08:44:45.838769 8495 tablet_bootstrap.cc:654] T 4d82c36f50f045aea0835d931324b9f8 P e4860201f08e45c98eea6befb1984108: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:45.839738 8494 tablet_bootstrap.cc:492] T 4d82c36f50f045aea0835d931324b9f8 P 6951c64acb48467bb84a75d1b715b8c1: No bootstrap required, opened a new log
I20260928 08:44:45.837925 8497 tablet_bootstrap.cc:492] T 4d82c36f50f045aea0835d931324b9f8 P 479cdfbbfa0a496abd505220830d412e: Bootstrap starting.
I20260928 08:44:45.839979 8495 log.cc:824] T 4d82c36f50f045aea0835d931324b9f8 P e4860201f08e45c98eea6befb1984108: Log is configured to *not* fsync() on all Append() calls
I20260928 08:44:45.840049 8494 ts_tablet_manager.cc:1397] T 4d82c36f50f045aea0835d931324b9f8 P 6951c64acb48467bb84a75d1b715b8c1: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260928 08:44:45.843257 8495 tablet_bootstrap.cc:492] T 4d82c36f50f045aea0835d931324b9f8 P e4860201f08e45c98eea6befb1984108: No bootstrap required, opened a new log
I20260928 08:44:45.843428 8497 tablet_bootstrap.cc:654] T 4d82c36f50f045aea0835d931324b9f8 P 479cdfbbfa0a496abd505220830d412e: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:45.843516 8495 ts_tablet_manager.cc:1397] T 4d82c36f50f045aea0835d931324b9f8 P e4860201f08e45c98eea6befb1984108: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 08:44:45.844583 8497 log.cc:824] T 4d82c36f50f045aea0835d931324b9f8 P 479cdfbbfa0a496abd505220830d412e: Log is configured to *not* fsync() on all Append() calls
I20260928 08:44:45.847295 8497 tablet_bootstrap.cc:492] T 4d82c36f50f045aea0835d931324b9f8 P 479cdfbbfa0a496abd505220830d412e: No bootstrap required, opened a new log
I20260928 08:44:45.847607 8497 ts_tablet_manager.cc:1397] T 4d82c36f50f045aea0835d931324b9f8 P 479cdfbbfa0a496abd505220830d412e: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260928 08:44:45.850449 8494 raft_consensus.cc:348] T 4d82c36f50f045aea0835d931324b9f8 P 6951c64acb48467bb84a75d1b715b8c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } }
I20260928 08:44:45.850888 8494 raft_consensus.cc:374] T 4d82c36f50f045aea0835d931324b9f8 P 6951c64acb48467bb84a75d1b715b8c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:45.851017 8494 raft_consensus.cc:729] T 4d82c36f50f045aea0835d931324b9f8 P 6951c64acb48467bb84a75d1b715b8c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6951c64acb48467bb84a75d1b715b8c1, State: Initialized, Role: FOLLOWER
I20260928 08:44:45.851642 8494 consensus_queue.cc:260] T 4d82c36f50f045aea0835d931324b9f8 P 6951c64acb48467bb84a75d1b715b8c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } }
I20260928 08:44:45.854687 8495 raft_consensus.cc:348] T 4d82c36f50f045aea0835d931324b9f8 P e4860201f08e45c98eea6befb1984108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } }
I20260928 08:44:45.855283 8495 raft_consensus.cc:374] T 4d82c36f50f045aea0835d931324b9f8 P e4860201f08e45c98eea6befb1984108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:45.855425 8495 raft_consensus.cc:729] T 4d82c36f50f045aea0835d931324b9f8 P e4860201f08e45c98eea6befb1984108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4860201f08e45c98eea6befb1984108, State: Initialized, Role: FOLLOWER
I20260928 08:44:45.856212 8495 consensus_queue.cc:260] T 4d82c36f50f045aea0835d931324b9f8 P e4860201f08e45c98eea6befb1984108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } }
I20260928 08:44:45.856469 8497 raft_consensus.cc:348] T 4d82c36f50f045aea0835d931324b9f8 P 479cdfbbfa0a496abd505220830d412e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } }
I20260928 08:44:45.856985 8497 raft_consensus.cc:374] T 4d82c36f50f045aea0835d931324b9f8 P 479cdfbbfa0a496abd505220830d412e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:45.857129 8497 raft_consensus.cc:729] T 4d82c36f50f045aea0835d931324b9f8 P 479cdfbbfa0a496abd505220830d412e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 479cdfbbfa0a496abd505220830d412e, State: Initialized, Role: FOLLOWER
I20260928 08:44:45.858446 8495 ts_tablet_manager.cc:1428] T 4d82c36f50f045aea0835d931324b9f8 P e4860201f08e45c98eea6befb1984108: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
I20260928 08:44:45.859036 8497 consensus_queue.cc:260] T 4d82c36f50f045aea0835d931324b9f8 P 479cdfbbfa0a496abd505220830d412e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } }
I20260928 08:44:45.859513 8470 heartbeater.cc:503] Master 127.0.97.62:46411 was elected leader, sending a full tablet report...
I20260928 08:44:45.860143 8494 ts_tablet_manager.cc:1428] T 4d82c36f50f045aea0835d931324b9f8 P 6951c64acb48467bb84a75d1b715b8c1: Time spent starting tablet: real 0.020s user 0.016s sys 0.004s
I20260928 08:44:45.861032 8331 heartbeater.cc:503] Master 127.0.97.62:46411 was elected leader, sending a full tablet report...
I20260928 08:44:45.861829 8497 ts_tablet_manager.cc:1428] T 4d82c36f50f045aea0835d931324b9f8 P 479cdfbbfa0a496abd505220830d412e: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
W20260928 08:44:45.903986 8193 tablet.cc:2369] T 4d82c36f50f045aea0835d931324b9f8 P e4860201f08e45c98eea6befb1984108: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:44:45.927898 8332 tablet.cc:2369] T 4d82c36f50f045aea0835d931324b9f8 P 479cdfbbfa0a496abd505220830d412e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:44:45.934332 8502 raft_consensus.cc:482] T 4d82c36f50f045aea0835d931324b9f8 P 479cdfbbfa0a496abd505220830d412e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:44:45.934652 8502 raft_consensus.cc:504] T 4d82c36f50f045aea0835d931324b9f8 P 479cdfbbfa0a496abd505220830d412e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } }
I20260928 08:44:45.936679 8502 leader_election.cc:288] T 4d82c36f50f045aea0835d931324b9f8 P 479cdfbbfa0a496abd505220830d412e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4860201f08e45c98eea6befb1984108 (127.0.97.1:40955), 6951c64acb48467bb84a75d1b715b8c1 (127.0.97.3:34497)
I20260928 08:44:45.945199 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:45.937074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38727 (local address 127.0.97.1:40955)
0928 08:44:45.937306 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:45.937316 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:45.937828 (+ 512us) server_negotiation.cc:407] Connection header received
0928 08:44:45.937999 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:45.938007 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:45.938107 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:45.938214 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:45.938868 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.939954 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.942682 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.944071 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.944213 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:45.944313 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:45.944819 (+ 506us) server_negotiation.cc:299] Negotiation successful
0928 08:44:45.944959 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 08:44:45.945206 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:45.937081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40975 (local address 127.0.97.3:34497)
0928 08:44:45.937331 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:45.937340 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:45.938468 (+ 1128us) server_negotiation.cc:407] Connection header received
0928 08:44:45.938605 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:45.938610 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:45.938676 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:45.938750 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:45.939216 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.940225 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.942682 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.944008 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.944193 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:45.944311 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:45.944819 (+ 508us) server_negotiation.cc:299] Negotiation successful
0928 08:44:45.944952 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 08:44:45.945379 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:45.937073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40955 (local address 127.0.97.2:38727)
0928 08:44:45.937712 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:45.937729 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:45.937855 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:45.938247 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:45.938253 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:45.938283 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:45.938685 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:45.938694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.940089 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.940093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:45.942488 (+ 2395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:45.942499 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.944326 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.944331 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:45.944564 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:45.944674 (+ 110us) client_negotiation.cc:769] Sending connection context
0928 08:44:45.944752 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 08:44:45.944936 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"mutex_wait_us":119,"thread_start_us":316,"threads_started":1}
I20260928 08:44:45.945700 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:45.937639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34497 (local address 127.0.97.2:40975)
0928 08:44:45.938350 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:45.938367 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:45.938490 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:45.938789 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:45.938793 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:45.938819 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:45.939064 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:45.939071 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.940337 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.940344 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:45.942486 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:45.942499 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:45.944313 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:45.944320 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:45.944566 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:45.944648 (+ 82us) client_negotiation.cc:769] Sending connection context
0928 08:44:45.944752 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 08:44:45.944936 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"mutex_wait_us":107,"thread_start_us":391,"threads_started":1}
I20260928 08:44:45.946528 8142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d82c36f50f045aea0835d931324b9f8" candidate_uuid: "479cdfbbfa0a496abd505220830d412e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4860201f08e45c98eea6befb1984108" is_pre_election: true
I20260928 08:44:45.946785 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d82c36f50f045aea0835d931324b9f8" candidate_uuid: "479cdfbbfa0a496abd505220830d412e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6951c64acb48467bb84a75d1b715b8c1" is_pre_election: true
I20260928 08:44:45.947183 8142 raft_consensus.cc:2454] T 4d82c36f50f045aea0835d931324b9f8 P e4860201f08e45c98eea6befb1984108 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 479cdfbbfa0a496abd505220830d412e in term 0.
I20260928 08:44:45.947285 8425 raft_consensus.cc:2454] T 4d82c36f50f045aea0835d931324b9f8 P 6951c64acb48467bb84a75d1b715b8c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 479cdfbbfa0a496abd505220830d412e in term 0.
I20260928 08:44:45.947990 8215 leader_election.cc:302] T 4d82c36f50f045aea0835d931324b9f8 P 479cdfbbfa0a496abd505220830d412e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 479cdfbbfa0a496abd505220830d412e, 6951c64acb48467bb84a75d1b715b8c1; no voters:
I20260928 08:44:45.948460 8502 raft_consensus.cc:2781] T 4d82c36f50f045aea0835d931324b9f8 P 479cdfbbfa0a496abd505220830d412e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:44:45.948607 8502 raft_consensus.cc:482] T 4d82c36f50f045aea0835d931324b9f8 P 479cdfbbfa0a496abd505220830d412e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:44:45.948704 8502 raft_consensus.cc:3037] T 4d82c36f50f045aea0835d931324b9f8 P 479cdfbbfa0a496abd505220830d412e [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:45.950789 8502 raft_consensus.cc:504] T 4d82c36f50f045aea0835d931324b9f8 P 479cdfbbfa0a496abd505220830d412e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } }
I20260928 08:44:45.951406 8502 leader_election.cc:288] T 4d82c36f50f045aea0835d931324b9f8 P 479cdfbbfa0a496abd505220830d412e [CANDIDATE]: Term 1 election: Requested vote from peers e4860201f08e45c98eea6befb1984108 (127.0.97.1:40955), 6951c64acb48467bb84a75d1b715b8c1 (127.0.97.3:34497)
I20260928 08:44:45.951912 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d82c36f50f045aea0835d931324b9f8" candidate_uuid: "479cdfbbfa0a496abd505220830d412e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6951c64acb48467bb84a75d1b715b8c1"
I20260928 08:44:45.951912 8142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d82c36f50f045aea0835d931324b9f8" candidate_uuid: "479cdfbbfa0a496abd505220830d412e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4860201f08e45c98eea6befb1984108"
I20260928 08:44:45.952113 8142 raft_consensus.cc:3037] T 4d82c36f50f045aea0835d931324b9f8 P e4860201f08e45c98eea6befb1984108 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:45.952113 8425 raft_consensus.cc:3037] T 4d82c36f50f045aea0835d931324b9f8 P 6951c64acb48467bb84a75d1b715b8c1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:45.955368 8142 raft_consensus.cc:2454] T 4d82c36f50f045aea0835d931324b9f8 P e4860201f08e45c98eea6befb1984108 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 479cdfbbfa0a496abd505220830d412e in term 1.
W20260928 08:44:45.955397 8471 tablet.cc:2369] T 4d82c36f50f045aea0835d931324b9f8 P 6951c64acb48467bb84a75d1b715b8c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:44:45.955793 8212 leader_election.cc:302] T 4d82c36f50f045aea0835d931324b9f8 P 479cdfbbfa0a496abd505220830d412e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 479cdfbbfa0a496abd505220830d412e, e4860201f08e45c98eea6befb1984108; no voters:
I20260928 08:44:45.956118 8425 raft_consensus.cc:2454] T 4d82c36f50f045aea0835d931324b9f8 P 6951c64acb48467bb84a75d1b715b8c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 479cdfbbfa0a496abd505220830d412e in term 1.
I20260928 08:44:45.956473 8502 raft_consensus.cc:2781] T 4d82c36f50f045aea0835d931324b9f8 P 479cdfbbfa0a496abd505220830d412e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:44:45.956933 8502 raft_consensus.cc:686] T 4d82c36f50f045aea0835d931324b9f8 P 479cdfbbfa0a496abd505220830d412e [term 1 LEADER]: Becoming Leader. State: Replica: 479cdfbbfa0a496abd505220830d412e, State: Running, Role: LEADER
I20260928 08:44:45.957420 8502 consensus_queue.cc:237] T 4d82c36f50f045aea0835d931324b9f8 P 479cdfbbfa0a496abd505220830d412e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } }
I20260928 08:44:45.962280 7983 catalog_manager.cc:5495] T 4d82c36f50f045aea0835d931324b9f8 P 479cdfbbfa0a496abd505220830d412e reported cstate change: term changed from 0 to 1, leader changed from <none> to 479cdfbbfa0a496abd505220830d412e (127.0.97.2). New cstate: current_term: 1 leader_uuid: "479cdfbbfa0a496abd505220830d412e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } health_report { overall_health: HEALTHY } } }
I20260928 08:44:46.006021 8492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:46.000597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53276 (local address 127.0.97.2:40415)
0928 08:44:46.000848 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:46.000857 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:46.000944 (+ 87us) server_negotiation.cc:407] Connection header received
0928 08:44:46.001191 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:46.001195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:46.001272 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:46.001400 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:44:46.001732 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:46.002496 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:46.003787 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:46.004289 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:46.004399 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:46.004810 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:44:46.005379 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:44:46.005502 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:46.005612 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 08:44:46.005686 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 08:44:46.015082 8425 raft_consensus.cc:1261] T 4d82c36f50f045aea0835d931324b9f8 P 6951c64acb48467bb84a75d1b715b8c1 [term 1 FOLLOWER]: Refusing update from remote peer 479cdfbbfa0a496abd505220830d412e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:44:46.015082 8142 raft_consensus.cc:1261] T 4d82c36f50f045aea0835d931324b9f8 P e4860201f08e45c98eea6befb1984108 [term 1 FOLLOWER]: Refusing update from remote peer 479cdfbbfa0a496abd505220830d412e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:44:46.016145 8505 consensus_queue.cc:1035] T 4d82c36f50f045aea0835d931324b9f8 P 479cdfbbfa0a496abd505220830d412e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:44:46.016542 8502 consensus_queue.cc:1035] T 4d82c36f50f045aea0835d931324b9f8 P 479cdfbbfa0a496abd505220830d412e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:44:46.032814 8508 mvcc.cc:205] Tried to move back new op lower bound from 7334236512306950144 to 7334236512085417984. Current Snapshot: MvccSnapshot[applied={T|T < 7334236512306950144}]
I20260928 08:44:46.051488 7985 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:44:46.051779 7985 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:44:46.055320 7985 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef140c385b0b49229ff2e2ddd2a91894: Sending DeleteTablet for 3 replicas of tablet 4d82c36f50f045aea0835d931324b9f8
I20260928 08:44:46.056514 8122 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d82c36f50f045aea0835d931324b9f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:46 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46096
I20260928 08:44:46.056468 8400 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d82c36f50f045aea0835d931324b9f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:46 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37942
I20260928 08:44:46.057001 8261 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d82c36f50f045aea0835d931324b9f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:46 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53268
I20260928 08:44:46.058725 7985 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:44:46.059600 7985 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:44:46.070248 8121 tablet_service.cc:1467] Processing CreateTablet for tablet cde7715a4b6a4911a2cc7d10b53a77c8 (DEFAULT_TABLE table=test-table [id=fd3371f2879848a08465ff342ae037e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:46.070833 8121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cde7715a4b6a4911a2cc7d10b53a77c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:46.070943 8260 tablet_service.cc:1467] Processing CreateTablet for tablet cde7715a4b6a4911a2cc7d10b53a77c8 (DEFAULT_TABLE table=test-table [id=fd3371f2879848a08465ff342ae037e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:46.071439 8260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cde7715a4b6a4911a2cc7d10b53a77c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:46.072343 8399 tablet_service.cc:1467] Processing CreateTablet for tablet cde7715a4b6a4911a2cc7d10b53a77c8 (DEFAULT_TABLE table=test-table [id=fd3371f2879848a08465ff342ae037e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:46.072909 8399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cde7715a4b6a4911a2cc7d10b53a77c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:46.077956 8497 tablet_bootstrap.cc:492] T cde7715a4b6a4911a2cc7d10b53a77c8 P 479cdfbbfa0a496abd505220830d412e: Bootstrap starting.
I20260928 08:44:46.079596 8497 tablet_bootstrap.cc:654] T cde7715a4b6a4911a2cc7d10b53a77c8 P 479cdfbbfa0a496abd505220830d412e: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:46.086513 8495 tablet_bootstrap.cc:492] T cde7715a4b6a4911a2cc7d10b53a77c8 P e4860201f08e45c98eea6befb1984108: Bootstrap starting.
I20260928 08:44:46.087006 8494 tablet_bootstrap.cc:492] T cde7715a4b6a4911a2cc7d10b53a77c8 P 6951c64acb48467bb84a75d1b715b8c1: Bootstrap starting.
I20260928 08:44:46.088146 8495 tablet_bootstrap.cc:654] T cde7715a4b6a4911a2cc7d10b53a77c8 P e4860201f08e45c98eea6befb1984108: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:46.088738 8494 tablet_bootstrap.cc:654] T cde7715a4b6a4911a2cc7d10b53a77c8 P 6951c64acb48467bb84a75d1b715b8c1: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:46.099313 8497 tablet_bootstrap.cc:492] T cde7715a4b6a4911a2cc7d10b53a77c8 P 479cdfbbfa0a496abd505220830d412e: No bootstrap required, opened a new log
I20260928 08:44:46.099573 8497 ts_tablet_manager.cc:1397] T cde7715a4b6a4911a2cc7d10b53a77c8 P 479cdfbbfa0a496abd505220830d412e: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.010s
I20260928 08:44:46.100538 8497 raft_consensus.cc:348] T cde7715a4b6a4911a2cc7d10b53a77c8 P 479cdfbbfa0a496abd505220830d412e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } }
I20260928 08:44:46.101442 8497 raft_consensus.cc:374] T cde7715a4b6a4911a2cc7d10b53a77c8 P 479cdfbbfa0a496abd505220830d412e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:46.102072 8497 raft_consensus.cc:729] T cde7715a4b6a4911a2cc7d10b53a77c8 P 479cdfbbfa0a496abd505220830d412e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 479cdfbbfa0a496abd505220830d412e, State: Initialized, Role: FOLLOWER
I20260928 08:44:46.102593 8497 consensus_queue.cc:260] T cde7715a4b6a4911a2cc7d10b53a77c8 P 479cdfbbfa0a496abd505220830d412e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } }
I20260928 08:44:46.103716 8497 ts_tablet_manager.cc:1428] T cde7715a4b6a4911a2cc7d10b53a77c8 P 479cdfbbfa0a496abd505220830d412e: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260928 08:44:46.101963 8521 tablet_replica.cc:330] T 4d82c36f50f045aea0835d931324b9f8 P 479cdfbbfa0a496abd505220830d412e: stopping tablet replica
I20260928 08:44:46.104874 8521 raft_consensus.cc:2229] T 4d82c36f50f045aea0835d931324b9f8 P 479cdfbbfa0a496abd505220830d412e [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:44:46.105602 8521 raft_consensus.cc:2258] T 4d82c36f50f045aea0835d931324b9f8 P 479cdfbbfa0a496abd505220830d412e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:46.108475 8521 ts_tablet_manager.cc:1905] T 4d82c36f50f045aea0835d931324b9f8 P 479cdfbbfa0a496abd505220830d412e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:46.111320 8495 tablet_bootstrap.cc:492] T cde7715a4b6a4911a2cc7d10b53a77c8 P e4860201f08e45c98eea6befb1984108: No bootstrap required, opened a new log
I20260928 08:44:46.111550 8495 ts_tablet_manager.cc:1397] T cde7715a4b6a4911a2cc7d10b53a77c8 P e4860201f08e45c98eea6befb1984108: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.000s
I20260928 08:44:46.112476 8495 raft_consensus.cc:348] T cde7715a4b6a4911a2cc7d10b53a77c8 P e4860201f08e45c98eea6befb1984108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } }
I20260928 08:44:46.112749 8495 raft_consensus.cc:374] T cde7715a4b6a4911a2cc7d10b53a77c8 P e4860201f08e45c98eea6befb1984108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:46.112890 8495 raft_consensus.cc:729] T cde7715a4b6a4911a2cc7d10b53a77c8 P e4860201f08e45c98eea6befb1984108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4860201f08e45c98eea6befb1984108, State: Initialized, Role: FOLLOWER
I20260928 08:44:46.113262 8495 consensus_queue.cc:260] T cde7715a4b6a4911a2cc7d10b53a77c8 P e4860201f08e45c98eea6befb1984108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } }
I20260928 08:44:46.115708 8521 ts_tablet_manager.cc:1918] T 4d82c36f50f045aea0835d931324b9f8 P 479cdfbbfa0a496abd505220830d412e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:44:46.115931 8521 log.cc:1196] T 4d82c36f50f045aea0835d931324b9f8 P 479cdfbbfa0a496abd505220830d412e: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-1/wal/wals/4d82c36f50f045aea0835d931324b9f8
I20260928 08:44:46.116516 8521 ts_tablet_manager.cc:1939] T 4d82c36f50f045aea0835d931324b9f8 P 479cdfbbfa0a496abd505220830d412e: Deleting consensus metadata
I20260928 08:44:46.116739 8495 ts_tablet_manager.cc:1428] T cde7715a4b6a4911a2cc7d10b53a77c8 P e4860201f08e45c98eea6befb1984108: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260928 08:44:46.118396 7969 catalog_manager.cc:4841] TS 479cdfbbfa0a496abd505220830d412e (127.0.97.2:40415): tablet 4d82c36f50f045aea0835d931324b9f8 (table test-table [id=2b56e1c15cf349e6bdd62403f03aa494]) successfully deleted
I20260928 08:44:46.119513 8519 tablet_replica.cc:330] T 4d82c36f50f045aea0835d931324b9f8 P e4860201f08e45c98eea6befb1984108: stopping tablet replica
I20260928 08:44:46.119890 8494 tablet_bootstrap.cc:492] T cde7715a4b6a4911a2cc7d10b53a77c8 P 6951c64acb48467bb84a75d1b715b8c1: No bootstrap required, opened a new log
I20260928 08:44:46.119957 8519 raft_consensus.cc:2229] T 4d82c36f50f045aea0835d931324b9f8 P e4860201f08e45c98eea6befb1984108 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:44:46.120055 8494 ts_tablet_manager.cc:1397] T cde7715a4b6a4911a2cc7d10b53a77c8 P 6951c64acb48467bb84a75d1b715b8c1: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.009s
I20260928 08:44:46.120404 8519 raft_consensus.cc:2258] T 4d82c36f50f045aea0835d931324b9f8 P e4860201f08e45c98eea6befb1984108 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:46.120883 8494 raft_consensus.cc:348] T cde7715a4b6a4911a2cc7d10b53a77c8 P 6951c64acb48467bb84a75d1b715b8c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } }
I20260928 08:44:46.121182 8494 raft_consensus.cc:374] T cde7715a4b6a4911a2cc7d10b53a77c8 P 6951c64acb48467bb84a75d1b715b8c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:46.121287 8494 raft_consensus.cc:729] T cde7715a4b6a4911a2cc7d10b53a77c8 P 6951c64acb48467bb84a75d1b715b8c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6951c64acb48467bb84a75d1b715b8c1, State: Initialized, Role: FOLLOWER
I20260928 08:44:46.121304 8520 tablet_replica.cc:330] T 4d82c36f50f045aea0835d931324b9f8 P 6951c64acb48467bb84a75d1b715b8c1: stopping tablet replica
I20260928 08:44:46.121558 8494 consensus_queue.cc:260] T cde7715a4b6a4911a2cc7d10b53a77c8 P 6951c64acb48467bb84a75d1b715b8c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } }
I20260928 08:44:46.121786 8520 raft_consensus.cc:2229] T 4d82c36f50f045aea0835d931324b9f8 P 6951c64acb48467bb84a75d1b715b8c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:44:46.122095 8494 ts_tablet_manager.cc:1428] T cde7715a4b6a4911a2cc7d10b53a77c8 P 6951c64acb48467bb84a75d1b715b8c1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 08:44:46.122243 8520 raft_consensus.cc:2258] T 4d82c36f50f045aea0835d931324b9f8 P 6951c64acb48467bb84a75d1b715b8c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:46.123260 8519 ts_tablet_manager.cc:1905] T 4d82c36f50f045aea0835d931324b9f8 P e4860201f08e45c98eea6befb1984108: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:46.124536 8520 ts_tablet_manager.cc:1905] T 4d82c36f50f045aea0835d931324b9f8 P 6951c64acb48467bb84a75d1b715b8c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:46.129791 8519 ts_tablet_manager.cc:1918] T 4d82c36f50f045aea0835d931324b9f8 P e4860201f08e45c98eea6befb1984108: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:44:46.129961 8519 log.cc:1196] T 4d82c36f50f045aea0835d931324b9f8 P e4860201f08e45c98eea6befb1984108: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-0/wal/wals/4d82c36f50f045aea0835d931324b9f8
I20260928 08:44:46.130437 8519 ts_tablet_manager.cc:1939] T 4d82c36f50f045aea0835d931324b9f8 P e4860201f08e45c98eea6befb1984108: Deleting consensus metadata
I20260928 08:44:46.131361 8520 ts_tablet_manager.cc:1918] T 4d82c36f50f045aea0835d931324b9f8 P 6951c64acb48467bb84a75d1b715b8c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:44:46.131525 8520 log.cc:1196] T 4d82c36f50f045aea0835d931324b9f8 P 6951c64acb48467bb84a75d1b715b8c1: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-2/wal/wals/4d82c36f50f045aea0835d931324b9f8
I20260928 08:44:46.132045 7969 catalog_manager.cc:4841] TS e4860201f08e45c98eea6befb1984108 (127.0.97.1:40955): tablet 4d82c36f50f045aea0835d931324b9f8 (table test-table [id=2b56e1c15cf349e6bdd62403f03aa494]) successfully deleted
I20260928 08:44:46.132244 8520 ts_tablet_manager.cc:1939] T 4d82c36f50f045aea0835d931324b9f8 P 6951c64acb48467bb84a75d1b715b8c1: Deleting consensus metadata
I20260928 08:44:46.133839 7972 catalog_manager.cc:4841] TS 6951c64acb48467bb84a75d1b715b8c1 (127.0.97.3:34497): tablet 4d82c36f50f045aea0835d931324b9f8 (table test-table [id=2b56e1c15cf349e6bdd62403f03aa494]) successfully deleted
I20260928 08:44:46.263042 8505 raft_consensus.cc:482] T cde7715a4b6a4911a2cc7d10b53a77c8 P 479cdfbbfa0a496abd505220830d412e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:44:46.263309 8505 raft_consensus.cc:504] T cde7715a4b6a4911a2cc7d10b53a77c8 P 479cdfbbfa0a496abd505220830d412e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } }
I20260928 08:44:46.264137 8505 leader_election.cc:288] T cde7715a4b6a4911a2cc7d10b53a77c8 P 479cdfbbfa0a496abd505220830d412e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4860201f08e45c98eea6befb1984108 (127.0.97.1:40955), 6951c64acb48467bb84a75d1b715b8c1 (127.0.97.3:34497)
I20260928 08:44:46.264673 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cde7715a4b6a4911a2cc7d10b53a77c8" candidate_uuid: "479cdfbbfa0a496abd505220830d412e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6951c64acb48467bb84a75d1b715b8c1" is_pre_election: true
I20260928 08:44:46.264726 8142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cde7715a4b6a4911a2cc7d10b53a77c8" candidate_uuid: "479cdfbbfa0a496abd505220830d412e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4860201f08e45c98eea6befb1984108" is_pre_election: true
I20260928 08:44:46.264971 8425 raft_consensus.cc:2454] T cde7715a4b6a4911a2cc7d10b53a77c8 P 6951c64acb48467bb84a75d1b715b8c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 479cdfbbfa0a496abd505220830d412e in term 0.
I20260928 08:44:46.264979 8142 raft_consensus.cc:2454] T cde7715a4b6a4911a2cc7d10b53a77c8 P e4860201f08e45c98eea6befb1984108 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 479cdfbbfa0a496abd505220830d412e in term 0.
I20260928 08:44:46.265491 8215 leader_election.cc:302] T cde7715a4b6a4911a2cc7d10b53a77c8 P 479cdfbbfa0a496abd505220830d412e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 479cdfbbfa0a496abd505220830d412e, 6951c64acb48467bb84a75d1b715b8c1; no voters:
I20260928 08:44:46.265818 8505 raft_consensus.cc:2781] T cde7715a4b6a4911a2cc7d10b53a77c8 P 479cdfbbfa0a496abd505220830d412e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:44:46.265926 8505 raft_consensus.cc:482] T cde7715a4b6a4911a2cc7d10b53a77c8 P 479cdfbbfa0a496abd505220830d412e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:44:46.266028 8505 raft_consensus.cc:3037] T cde7715a4b6a4911a2cc7d10b53a77c8 P 479cdfbbfa0a496abd505220830d412e [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:46.268491 8505 raft_consensus.cc:504] T cde7715a4b6a4911a2cc7d10b53a77c8 P 479cdfbbfa0a496abd505220830d412e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } }
I20260928 08:44:46.269073 8505 leader_election.cc:288] T cde7715a4b6a4911a2cc7d10b53a77c8 P 479cdfbbfa0a496abd505220830d412e [CANDIDATE]: Term 1 election: Requested vote from peers e4860201f08e45c98eea6befb1984108 (127.0.97.1:40955), 6951c64acb48467bb84a75d1b715b8c1 (127.0.97.3:34497)
I20260928 08:44:46.269471 8142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cde7715a4b6a4911a2cc7d10b53a77c8" candidate_uuid: "479cdfbbfa0a496abd505220830d412e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4860201f08e45c98eea6befb1984108"
I20260928 08:44:46.269614 8425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cde7715a4b6a4911a2cc7d10b53a77c8" candidate_uuid: "479cdfbbfa0a496abd505220830d412e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6951c64acb48467bb84a75d1b715b8c1"
I20260928 08:44:46.269713 8142 raft_consensus.cc:3037] T cde7715a4b6a4911a2cc7d10b53a77c8 P e4860201f08e45c98eea6befb1984108 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:46.269815 8425 raft_consensus.cc:3037] T cde7715a4b6a4911a2cc7d10b53a77c8 P 6951c64acb48467bb84a75d1b715b8c1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:46.272663 8142 raft_consensus.cc:2454] T cde7715a4b6a4911a2cc7d10b53a77c8 P e4860201f08e45c98eea6befb1984108 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 479cdfbbfa0a496abd505220830d412e in term 1.
I20260928 08:44:46.272663 8425 raft_consensus.cc:2454] T cde7715a4b6a4911a2cc7d10b53a77c8 P 6951c64acb48467bb84a75d1b715b8c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 479cdfbbfa0a496abd505220830d412e in term 1.
I20260928 08:44:46.273111 8215 leader_election.cc:302] T cde7715a4b6a4911a2cc7d10b53a77c8 P 479cdfbbfa0a496abd505220830d412e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 479cdfbbfa0a496abd505220830d412e, 6951c64acb48467bb84a75d1b715b8c1; no voters:
I20260928 08:44:46.273504 8505 raft_consensus.cc:2781] T cde7715a4b6a4911a2cc7d10b53a77c8 P 479cdfbbfa0a496abd505220830d412e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:44:46.273698 8505 raft_consensus.cc:686] T cde7715a4b6a4911a2cc7d10b53a77c8 P 479cdfbbfa0a496abd505220830d412e [term 1 LEADER]: Becoming Leader. State: Replica: 479cdfbbfa0a496abd505220830d412e, State: Running, Role: LEADER
I20260928 08:44:46.273944 8505 consensus_queue.cc:237] T cde7715a4b6a4911a2cc7d10b53a77c8 P 479cdfbbfa0a496abd505220830d412e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } }
I20260928 08:44:46.276651 7983 catalog_manager.cc:5495] T cde7715a4b6a4911a2cc7d10b53a77c8 P 479cdfbbfa0a496abd505220830d412e reported cstate change: term changed from 0 to 1, leader changed from <none> to 479cdfbbfa0a496abd505220830d412e (127.0.97.2). New cstate: current_term: 1 leader_uuid: "479cdfbbfa0a496abd505220830d412e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } health_report { overall_health: UNKNOWN } } }
Sep 28 08:44:46 dist-test-slave-n6xs krb5kdc[7940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585083, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.62@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 08:44:46.351471 8531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:46.338014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35534 (local address 127.0.97.62:46411)
0928 08:44:46.338610 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:46.338620 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:46.338865 (+ 245us) server_negotiation.cc:407] Connection header received
0928 08:44:46.339084 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:46.339093 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:46.339154 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:46.339259 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:46.340332 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:46.341069 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:46.342270 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:46.342661 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:46.342796 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:46.346111 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:46.346116 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:46.346119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:46.346146 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:46.348903 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:46.349548 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:46.349555 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:46.349561 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:46.349678 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:46.350052 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:46.350059 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:46.350062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:46.350309 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:46.350459 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:46.350839 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 08:44:46.351043 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":426,"thread_start_us":297,"threads_started":1}
I20260928 08:44:46.351531 8532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:46.337987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46411 (local address 127.0.0.1:35534)
0928 08:44:46.338761 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:46.338776 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:46.338945 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:46.339296 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:46.339301 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:46.340025 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:46.340196 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:46.340204 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:46.341207 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:46.341213 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:46.342137 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:46.342147 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:46.342819 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:46.342824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:46.343000 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:46.343689 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:46.343710 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:46.345911 (+ 2201us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:46.349029 (+ 3118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:46.349043 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:46.349051 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:46.349436 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:46.349781 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:46.349788 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:46.349800 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:46.349947 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:46.350427 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:46.350437 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:46.350593 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 08:44:46.350758 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 08:44:46.351167 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"mutex_wait_us":127,"thread_start_us":523,"threads_started":1}
I20260928 08:44:46.357309 7983 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:35534:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 08:44:46.358275 7983 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:44:46.365365 8400 tablet_service.cc:1467] Processing CreateTablet for tablet d829d6113b8c47f8a6e8acc3adc640ba (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2de4d217de3048e7945ef40e627d15bf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:44:46.365252 8261 tablet_service.cc:1467] Processing CreateTablet for tablet d829d6113b8c47f8a6e8acc3adc640ba (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2de4d217de3048e7945ef40e627d15bf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:44:46.365921 8261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d829d6113b8c47f8a6e8acc3adc640ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:46.365912 8400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d829d6113b8c47f8a6e8acc3adc640ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:46.365787 8122 tablet_service.cc:1467] Processing CreateTablet for tablet d829d6113b8c47f8a6e8acc3adc640ba (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2de4d217de3048e7945ef40e627d15bf]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 08:44:46.366323 8122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d829d6113b8c47f8a6e8acc3adc640ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:46.373062 8497 tablet_bootstrap.cc:492] T d829d6113b8c47f8a6e8acc3adc640ba P 479cdfbbfa0a496abd505220830d412e: Bootstrap starting.
I20260928 08:44:46.373221 8494 tablet_bootstrap.cc:492] T d829d6113b8c47f8a6e8acc3adc640ba P 6951c64acb48467bb84a75d1b715b8c1: Bootstrap starting.
I20260928 08:44:46.374845 8494 tablet_bootstrap.cc:654] T d829d6113b8c47f8a6e8acc3adc640ba P 6951c64acb48467bb84a75d1b715b8c1: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:46.374840 8497 tablet_bootstrap.cc:654] T d829d6113b8c47f8a6e8acc3adc640ba P 479cdfbbfa0a496abd505220830d412e: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:46.375989 8495 tablet_bootstrap.cc:492] T d829d6113b8c47f8a6e8acc3adc640ba P e4860201f08e45c98eea6befb1984108: Bootstrap starting.
I20260928 08:44:46.378167 8495 tablet_bootstrap.cc:654] T d829d6113b8c47f8a6e8acc3adc640ba P e4860201f08e45c98eea6befb1984108: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:46.384541 8495 tablet_bootstrap.cc:492] T d829d6113b8c47f8a6e8acc3adc640ba P e4860201f08e45c98eea6befb1984108: No bootstrap required, opened a new log
I20260928 08:44:46.384868 8495 ts_tablet_manager.cc:1397] T d829d6113b8c47f8a6e8acc3adc640ba P e4860201f08e45c98eea6befb1984108: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260928 08:44:46.387344 8494 tablet_bootstrap.cc:492] T d829d6113b8c47f8a6e8acc3adc640ba P 6951c64acb48467bb84a75d1b715b8c1: No bootstrap required, opened a new log
I20260928 08:44:46.387529 8494 ts_tablet_manager.cc:1397] T d829d6113b8c47f8a6e8acc3adc640ba P 6951c64acb48467bb84a75d1b715b8c1: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.002s
I20260928 08:44:46.389252 8494 raft_consensus.cc:348] T d829d6113b8c47f8a6e8acc3adc640ba P 6951c64acb48467bb84a75d1b715b8c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } } peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } }
I20260928 08:44:46.389503 8494 raft_consensus.cc:374] T d829d6113b8c47f8a6e8acc3adc640ba P 6951c64acb48467bb84a75d1b715b8c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:46.389580 8494 raft_consensus.cc:729] T d829d6113b8c47f8a6e8acc3adc640ba P 6951c64acb48467bb84a75d1b715b8c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6951c64acb48467bb84a75d1b715b8c1, State: Initialized, Role: FOLLOWER
I20260928 08:44:46.389829 8494 consensus_queue.cc:260] T d829d6113b8c47f8a6e8acc3adc640ba P 6951c64acb48467bb84a75d1b715b8c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } } peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } }
I20260928 08:44:46.390410 8494 ts_tablet_manager.cc:1428] T d829d6113b8c47f8a6e8acc3adc640ba P 6951c64acb48467bb84a75d1b715b8c1: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260928 08:44:46.390296 8500 tablet_replica.cc:439] T d829d6113b8c47f8a6e8acc3adc640ba P 6951c64acb48467bb84a75d1b715b8c1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } } peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } }
I20260928 08:44:46.390630 8500 tablet_replica.cc:442] T d829d6113b8c47f8a6e8acc3adc640ba P 6951c64acb48467bb84a75d1b715b8c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:44:46.393066 8497 tablet_bootstrap.cc:492] T d829d6113b8c47f8a6e8acc3adc640ba P 479cdfbbfa0a496abd505220830d412e: No bootstrap required, opened a new log
I20260928 08:44:46.393290 8497 ts_tablet_manager.cc:1397] T d829d6113b8c47f8a6e8acc3adc640ba P 479cdfbbfa0a496abd505220830d412e: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.014s
I20260928 08:44:46.385910 8495 raft_consensus.cc:348] T d829d6113b8c47f8a6e8acc3adc640ba P e4860201f08e45c98eea6befb1984108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } } peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } }
I20260928 08:44:46.393894 8495 raft_consensus.cc:374] T d829d6113b8c47f8a6e8acc3adc640ba P e4860201f08e45c98eea6befb1984108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:46.393973 8495 raft_consensus.cc:729] T d829d6113b8c47f8a6e8acc3adc640ba P e4860201f08e45c98eea6befb1984108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4860201f08e45c98eea6befb1984108, State: Initialized, Role: FOLLOWER
I20260928 08:44:46.394181 8495 consensus_queue.cc:260] T d829d6113b8c47f8a6e8acc3adc640ba P e4860201f08e45c98eea6befb1984108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } } peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } }
I20260928 08:44:46.394187 8497 raft_consensus.cc:348] T d829d6113b8c47f8a6e8acc3adc640ba P 479cdfbbfa0a496abd505220830d412e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } } peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } }
I20260928 08:44:46.394448 8497 raft_consensus.cc:374] T d829d6113b8c47f8a6e8acc3adc640ba P 479cdfbbfa0a496abd505220830d412e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:46.394541 8497 raft_consensus.cc:729] T d829d6113b8c47f8a6e8acc3adc640ba P 479cdfbbfa0a496abd505220830d412e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 479cdfbbfa0a496abd505220830d412e, State: Initialized, Role: FOLLOWER
I20260928 08:44:46.394646 8501 tablet_replica.cc:439] T d829d6113b8c47f8a6e8acc3adc640ba P e4860201f08e45c98eea6befb1984108: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } } peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } }
I20260928 08:44:46.394794 8497 consensus_queue.cc:260] T d829d6113b8c47f8a6e8acc3adc640ba P 479cdfbbfa0a496abd505220830d412e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } } peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } }
I20260928 08:44:46.394994 8501 tablet_replica.cc:442] T d829d6113b8c47f8a6e8acc3adc640ba P e4860201f08e45c98eea6befb1984108: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:44:46.395465 8497 ts_tablet_manager.cc:1428] T d829d6113b8c47f8a6e8acc3adc640ba P 479cdfbbfa0a496abd505220830d412e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260928 08:44:46.395707 8505 tablet_replica.cc:439] T d829d6113b8c47f8a6e8acc3adc640ba P 479cdfbbfa0a496abd505220830d412e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } } peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } }
I20260928 08:44:46.396034 8505 tablet_replica.cc:442] T d829d6113b8c47f8a6e8acc3adc640ba P 479cdfbbfa0a496abd505220830d412e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:44:46.396481 8495 ts_tablet_manager.cc:1428] T d829d6113b8c47f8a6e8acc3adc640ba P e4860201f08e45c98eea6befb1984108: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20260928 08:44:46.558660 8500 raft_consensus.cc:482] T d829d6113b8c47f8a6e8acc3adc640ba P 6951c64acb48467bb84a75d1b715b8c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:44:46.558934 8500 raft_consensus.cc:504] T d829d6113b8c47f8a6e8acc3adc640ba P 6951c64acb48467bb84a75d1b715b8c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } } peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } }
I20260928 08:44:46.560460 8500 leader_election.cc:288] T d829d6113b8c47f8a6e8acc3adc640ba P 6951c64acb48467bb84a75d1b715b8c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 479cdfbbfa0a496abd505220830d412e (127.0.97.2:40415), e4860201f08e45c98eea6befb1984108 (127.0.97.1:40955)
I20260928 08:44:46.566781 8537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:46.560743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56757 (local address 127.0.97.2:40415)
0928 08:44:46.561225 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:46.561233 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:46.561255 (+ 22us) server_negotiation.cc:407] Connection header received
0928 08:44:46.561450 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:46.561459 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:46.561562 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:46.561657 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:46.562426 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:46.563190 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:46.565003 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:46.565842 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:46.565984 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:46.566088 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:46.566346 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 08:44:46.566475 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":251,"threads_started":1}
I20260928 08:44:46.566851 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:46.560655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40415 (local address 127.0.97.3:56757)
0928 08:44:46.561098 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:46.561117 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:46.561266 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:46.561868 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:46.561873 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:46.561899 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:46.562145 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:46.562157 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:46.563318 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:46.563323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:46.564862 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:46.564873 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:46.565993 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:46.565998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:46.566187 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:46.566213 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 08:44:46.566323 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 08:44:46.566460 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"mutex_wait_us":37,"spinlock_wait_cycles":10368,"thread_start_us":267,"threads_started":1}
I20260928 08:44:46.567049 8538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:46.561328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40955 (local address 127.0.97.3:55133)
0928 08:44:46.561848 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:46.561866 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:46.561993 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:46.562293 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:46.562298 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:46.562322 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:46.562638 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:46.562646 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:46.563592 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:46.563597 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:46.565077 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:46.565088 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:46.566234 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:46.566240 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:46.566405 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:46.566425 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 08:44:46.566520 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 08:44:46.566640 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":21,"spinlock_wait_cycles":896,"thread_start_us":211,"threads_started":1}
I20260928 08:44:46.567305 8539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:46.561402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:55133 (local address 127.0.97.1:40955)
0928 08:44:46.562002 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:46.562008 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:46.562025 (+ 17us) server_negotiation.cc:407] Connection header received
0928 08:44:46.562090 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:46.562095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:46.562161 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:46.562255 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:46.562762 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:46.563461 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:46.565251 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:46.566063 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:46.566199 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:46.566300 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:46.566543 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 08:44:46.566669 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":356,"threads_started":1}
I20260928 08:44:46.567680 8281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d829d6113b8c47f8a6e8acc3adc640ba" candidate_uuid: "6951c64acb48467bb84a75d1b715b8c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "479cdfbbfa0a496abd505220830d412e" is_pre_election: true
I20260928 08:44:46.567989 8142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d829d6113b8c47f8a6e8acc3adc640ba" candidate_uuid: "6951c64acb48467bb84a75d1b715b8c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4860201f08e45c98eea6befb1984108" is_pre_election: true
I20260928 08:44:46.568123 8281 raft_consensus.cc:2454] T d829d6113b8c47f8a6e8acc3adc640ba P 479cdfbbfa0a496abd505220830d412e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6951c64acb48467bb84a75d1b715b8c1 in term 0.
I20260928 08:44:46.568246 8142 raft_consensus.cc:2454] T d829d6113b8c47f8a6e8acc3adc640ba P e4860201f08e45c98eea6befb1984108 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6951c64acb48467bb84a75d1b715b8c1 in term 0.
I20260928 08:44:46.568629 8351 leader_election.cc:302] T d829d6113b8c47f8a6e8acc3adc640ba P 6951c64acb48467bb84a75d1b715b8c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 479cdfbbfa0a496abd505220830d412e, 6951c64acb48467bb84a75d1b715b8c1; no voters:
I20260928 08:44:46.568969 8500 raft_consensus.cc:2781] T d829d6113b8c47f8a6e8acc3adc640ba P 6951c64acb48467bb84a75d1b715b8c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:44:46.569096 8500 raft_consensus.cc:482] T d829d6113b8c47f8a6e8acc3adc640ba P 6951c64acb48467bb84a75d1b715b8c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:44:46.569204 8500 raft_consensus.cc:3037] T d829d6113b8c47f8a6e8acc3adc640ba P 6951c64acb48467bb84a75d1b715b8c1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:46.571165 8500 raft_consensus.cc:504] T d829d6113b8c47f8a6e8acc3adc640ba P 6951c64acb48467bb84a75d1b715b8c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } } peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } }
I20260928 08:44:46.571659 8500 leader_election.cc:288] T d829d6113b8c47f8a6e8acc3adc640ba P 6951c64acb48467bb84a75d1b715b8c1 [CANDIDATE]: Term 1 election: Requested vote from peers 479cdfbbfa0a496abd505220830d412e (127.0.97.2:40415), e4860201f08e45c98eea6befb1984108 (127.0.97.1:40955)
I20260928 08:44:46.572127 8281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d829d6113b8c47f8a6e8acc3adc640ba" candidate_uuid: "6951c64acb48467bb84a75d1b715b8c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "479cdfbbfa0a496abd505220830d412e"
I20260928 08:44:46.572178 8142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d829d6113b8c47f8a6e8acc3adc640ba" candidate_uuid: "6951c64acb48467bb84a75d1b715b8c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4860201f08e45c98eea6befb1984108"
I20260928 08:44:46.572312 8281 raft_consensus.cc:3037] T d829d6113b8c47f8a6e8acc3adc640ba P 479cdfbbfa0a496abd505220830d412e [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:46.572388 8142 raft_consensus.cc:3037] T d829d6113b8c47f8a6e8acc3adc640ba P e4860201f08e45c98eea6befb1984108 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:46.574720 8142 raft_consensus.cc:2454] T d829d6113b8c47f8a6e8acc3adc640ba P e4860201f08e45c98eea6befb1984108 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6951c64acb48467bb84a75d1b715b8c1 in term 1.
I20260928 08:44:46.574846 8281 raft_consensus.cc:2454] T d829d6113b8c47f8a6e8acc3adc640ba P 479cdfbbfa0a496abd505220830d412e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6951c64acb48467bb84a75d1b715b8c1 in term 1.
I20260928 08:44:46.575111 8351 leader_election.cc:302] T d829d6113b8c47f8a6e8acc3adc640ba P 6951c64acb48467bb84a75d1b715b8c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6951c64acb48467bb84a75d1b715b8c1, e4860201f08e45c98eea6befb1984108; no voters:
I20260928 08:44:46.575398 8500 raft_consensus.cc:2781] T d829d6113b8c47f8a6e8acc3adc640ba P 6951c64acb48467bb84a75d1b715b8c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:44:46.575765 8500 raft_consensus.cc:686] T d829d6113b8c47f8a6e8acc3adc640ba P 6951c64acb48467bb84a75d1b715b8c1 [term 1 LEADER]: Becoming Leader. State: Replica: 6951c64acb48467bb84a75d1b715b8c1, State: Running, Role: LEADER
I20260928 08:44:46.576198 8500 consensus_queue.cc:237] T d829d6113b8c47f8a6e8acc3adc640ba P 6951c64acb48467bb84a75d1b715b8c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } } peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } }
I20260928 08:44:46.577948 8540 tablet_replica.cc:439] T d829d6113b8c47f8a6e8acc3adc640ba P 6951c64acb48467bb84a75d1b715b8c1: TxnStatusTablet state changed. Reason: New leader 6951c64acb48467bb84a75d1b715b8c1. Latest consensus state: current_term: 1 leader_uuid: "6951c64acb48467bb84a75d1b715b8c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } } peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } }
I20260928 08:44:46.578369 8540 tablet_replica.cc:442] T d829d6113b8c47f8a6e8acc3adc640ba P 6951c64acb48467bb84a75d1b715b8c1: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:44:46.579044 8542 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 08:44:46.579280 8542 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 08:44:46.579871 7985 catalog_manager.cc:5495] T d829d6113b8c47f8a6e8acc3adc640ba P 6951c64acb48467bb84a75d1b715b8c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6951c64acb48467bb84a75d1b715b8c1 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "6951c64acb48467bb84a75d1b715b8c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } health_report { overall_health: UNKNOWN } } }
I20260928 08:44:46.633988 8532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:46.627314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34497 (local address 127.0.0.1:37956)
0928 08:44:46.627639 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:46.627666 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:46.627888 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:46.628636 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:46.628640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:46.628650 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:44:46.628806 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:46.628814 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:46.630309 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:46.630313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:46.631306 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:46.631317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:46.632281 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:46.632285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:46.632416 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:46.632467 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:44:46.633482 (+ 1015us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:44:46.633494 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 08:44:46.633542 (+ 48us) client_negotiation.cc:240] Negotiation successful
0928 08:44:46.633645 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 08:44:46.634085 8545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:46.627258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37956 (local address 127.0.97.3:34497)
0928 08:44:46.628197 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:46.628203 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:46.628229 (+ 26us) server_negotiation.cc:407] Connection header received
0928 08:44:46.628333 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:46.628339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:46.628435 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:46.628612 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:44:46.628958 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:46.630184 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:46.631436 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:46.632124 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:46.632244 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:46.632679 (+ 435us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:44:46.633381 (+ 702us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:44:46.633516 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:46.633572 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 08:44:46.633654 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":815,"thread_start_us":699,"threads_started":1}
I20260928 08:44:46.638275 8142 raft_consensus.cc:1261] T d829d6113b8c47f8a6e8acc3adc640ba P e4860201f08e45c98eea6befb1984108 [term 1 FOLLOWER]: Refusing update from remote peer 6951c64acb48467bb84a75d1b715b8c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:44:46.638356 8281 raft_consensus.cc:1261] T d829d6113b8c47f8a6e8acc3adc640ba P 479cdfbbfa0a496abd505220830d412e [term 1 FOLLOWER]: Refusing update from remote peer 6951c64acb48467bb84a75d1b715b8c1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:44:46.638943 8500 consensus_queue.cc:1035] T d829d6113b8c47f8a6e8acc3adc640ba P 6951c64acb48467bb84a75d1b715b8c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:44:46.639271 8540 consensus_queue.cc:1035] T d829d6113b8c47f8a6e8acc3adc640ba P 6951c64acb48467bb84a75d1b715b8c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:44:46.646008 8501 tablet_replica.cc:439] T d829d6113b8c47f8a6e8acc3adc640ba P e4860201f08e45c98eea6befb1984108: TxnStatusTablet state changed. Reason: New leader 6951c64acb48467bb84a75d1b715b8c1. Latest consensus state: current_term: 1 leader_uuid: "6951c64acb48467bb84a75d1b715b8c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } } peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } }
I20260928 08:44:46.646297 8501 tablet_replica.cc:442] T d829d6113b8c47f8a6e8acc3adc640ba P e4860201f08e45c98eea6befb1984108: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:44:46.647938 8546 mvcc.cc:205] Tried to move back new op lower bound from 7334236514863251456 to 7334236514619559936. Current Snapshot: MvccSnapshot[applied={T|T < 7334236514863251456}]
I20260928 08:44:46.648478 8540 tablet_replica.cc:439] T d829d6113b8c47f8a6e8acc3adc640ba P 6951c64acb48467bb84a75d1b715b8c1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6951c64acb48467bb84a75d1b715b8c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } } peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } }
I20260928 08:44:46.648744 8540 tablet_replica.cc:442] T d829d6113b8c47f8a6e8acc3adc640ba P 6951c64acb48467bb84a75d1b715b8c1: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:44:46.649030 8547 mvcc.cc:205] Tried to move back new op lower bound from 7334236514863251456 to 7334236514619559936. Current Snapshot: MvccSnapshot[applied={T|T < 7334236514863251456}]
I20260928 08:44:46.653947 8501 tablet_replica.cc:439] T d829d6113b8c47f8a6e8acc3adc640ba P e4860201f08e45c98eea6befb1984108: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6951c64acb48467bb84a75d1b715b8c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } } peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } }
I20260928 08:44:46.654191 8501 tablet_replica.cc:442] T d829d6113b8c47f8a6e8acc3adc640ba P e4860201f08e45c98eea6befb1984108: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:44:46.659747 8505 tablet_replica.cc:439] T d829d6113b8c47f8a6e8acc3adc640ba P 479cdfbbfa0a496abd505220830d412e: TxnStatusTablet state changed. Reason: New leader 6951c64acb48467bb84a75d1b715b8c1. Latest consensus state: current_term: 1 leader_uuid: "6951c64acb48467bb84a75d1b715b8c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } } peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } }
I20260928 08:44:46.660079 8505 tablet_replica.cc:442] T d829d6113b8c47f8a6e8acc3adc640ba P 479cdfbbfa0a496abd505220830d412e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:44:46.666261 8540 tablet_replica.cc:439] T d829d6113b8c47f8a6e8acc3adc640ba P 6951c64acb48467bb84a75d1b715b8c1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6951c64acb48467bb84a75d1b715b8c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } } peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } }
I20260928 08:44:46.666522 8540 tablet_replica.cc:442] T d829d6113b8c47f8a6e8acc3adc640ba P 6951c64acb48467bb84a75d1b715b8c1: This TxnStatusTablet replica's current role is: LEADER
I20260928 08:44:46.678383 8505 tablet_replica.cc:439] T d829d6113b8c47f8a6e8acc3adc640ba P 479cdfbbfa0a496abd505220830d412e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6951c64acb48467bb84a75d1b715b8c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } } peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } }
I20260928 08:44:46.678668 8505 tablet_replica.cc:442] T d829d6113b8c47f8a6e8acc3adc640ba P 479cdfbbfa0a496abd505220830d412e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 08:44:46.698029 8545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:46.683610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39465 (local address 127.0.97.3:34497)
0928 08:44:46.683771 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:46.683777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:46.692482 (+ 8705us) server_negotiation.cc:407] Connection header received
0928 08:44:46.692682 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:46.692688 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:46.692779 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:46.692883 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:44:46.693442 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:46.694426 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:46.696412 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:46.696830 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:46.696922 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:46.697420 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:44:46.697515 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:44:46.697580 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:46.697756 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 08:44:46.697816 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 08:44:46.698338 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:46.683213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34497 (local address 127.0.97.2:39465)
0928 08:44:46.692352 (+ 9139us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:46.692372 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:46.692538 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:46.692914 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:46.692919 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:46.692933 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:44:46.693275 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:46.693297 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:46.694552 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:46.694557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:46.696264 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:46.696279 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:46.696994 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:46.697001 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:46.697173 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:46.697240 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:44:46.697607 (+ 367us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:44:46.697618 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 08:44:46.697703 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 08:44:46.697845 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8926,"thread_start_us":8790,"threads_started":1}
I20260928 08:44:46.708736 8425 raft_consensus.cc:1261] T cde7715a4b6a4911a2cc7d10b53a77c8 P 6951c64acb48467bb84a75d1b715b8c1 [term 1 FOLLOWER]: Refusing update from remote peer 479cdfbbfa0a496abd505220830d412e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:44:46.708724 8142 raft_consensus.cc:1261] T cde7715a4b6a4911a2cc7d10b53a77c8 P e4860201f08e45c98eea6befb1984108 [term 1 FOLLOWER]: Refusing update from remote peer 479cdfbbfa0a496abd505220830d412e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:44:46.709271 8515 consensus_queue.cc:1035] T cde7715a4b6a4911a2cc7d10b53a77c8 P 479cdfbbfa0a496abd505220830d412e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:44:46.709829 8515 consensus_queue.cc:1035] T cde7715a4b6a4911a2cc7d10b53a77c8 P 479cdfbbfa0a496abd505220830d412e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:44:46.770824 8537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:46.756448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60871 (local address 127.0.97.2:40415)
0928 08:44:46.756696 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:46.756705 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:46.764494 (+ 7789us) server_negotiation.cc:407] Connection header received
0928 08:44:46.764687 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:46.764694 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:46.764782 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:46.764891 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:44:46.765815 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:46.766693 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:46.768515 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:46.769111 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:46.769206 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:46.769962 (+ 756us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:44:46.770115 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:44:46.770186 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:46.770414 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 08:44:46.770494 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 08:44:46.770988 8562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:46.755770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40415 (local address 127.0.97.3:60871)
0928 08:44:46.764368 (+ 8598us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:46.764389 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:46.764550 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:46.765087 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:46.765093 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:46.765107 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:44:46.765573 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:46.765592 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:46.766842 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:46.766850 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:46.768309 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:46.768325 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:46.769433 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:46.769439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:46.769644 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:46.769733 (+ 89us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:44:46.770284 (+ 551us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:44:46.770292 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 08:44:46.770391 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 08:44:46.770535 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8355,"thread_start_us":8170,"threads_started":1}
I20260928 08:44:46.800976 7983 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:44:46.801234 7983 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:44:46.804378 7983 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef140c385b0b49229ff2e2ddd2a91894: Sending DeleteTablet for 3 replicas of tablet cde7715a4b6a4911a2cc7d10b53a77c8
I20260928 08:44:46.805262 8122 tablet_service.cc:1514] Processing DeleteTablet for tablet cde7715a4b6a4911a2cc7d10b53a77c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:46 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46096
I20260928 08:44:46.805310 8400 tablet_service.cc:1514] Processing DeleteTablet for tablet cde7715a4b6a4911a2cc7d10b53a77c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:46 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37942
I20260928 08:44:46.805497 8261 tablet_service.cc:1514] Processing DeleteTablet for tablet cde7715a4b6a4911a2cc7d10b53a77c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:46 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53268
I20260928 08:44:46.817723 8565 tablet_replica.cc:330] T cde7715a4b6a4911a2cc7d10b53a77c8 P e4860201f08e45c98eea6befb1984108: stopping tablet replica
I20260928 08:44:46.818338 8567 tablet_replica.cc:330] T cde7715a4b6a4911a2cc7d10b53a77c8 P 479cdfbbfa0a496abd505220830d412e: stopping tablet replica
I20260928 08:44:46.818740 8567 raft_consensus.cc:2229] T cde7715a4b6a4911a2cc7d10b53a77c8 P 479cdfbbfa0a496abd505220830d412e [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:44:46.819128 8565 raft_consensus.cc:2229] T cde7715a4b6a4911a2cc7d10b53a77c8 P e4860201f08e45c98eea6befb1984108 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:44:46.819666 8567 raft_consensus.cc:2258] T cde7715a4b6a4911a2cc7d10b53a77c8 P 479cdfbbfa0a496abd505220830d412e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:46.820060 8565 raft_consensus.cc:2258] T cde7715a4b6a4911a2cc7d10b53a77c8 P e4860201f08e45c98eea6befb1984108 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:46.820173 8566 tablet_replica.cc:330] T cde7715a4b6a4911a2cc7d10b53a77c8 P 6951c64acb48467bb84a75d1b715b8c1: stopping tablet replica
I20260928 08:44:46.820389 8566 raft_consensus.cc:2229] T cde7715a4b6a4911a2cc7d10b53a77c8 P 6951c64acb48467bb84a75d1b715b8c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:44:46.820647 8566 raft_consensus.cc:2258] T cde7715a4b6a4911a2cc7d10b53a77c8 P 6951c64acb48467bb84a75d1b715b8c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:46.821678 8565 ts_tablet_manager.cc:1905] T cde7715a4b6a4911a2cc7d10b53a77c8 P e4860201f08e45c98eea6befb1984108: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:46.821791 8566 ts_tablet_manager.cc:1905] T cde7715a4b6a4911a2cc7d10b53a77c8 P 6951c64acb48467bb84a75d1b715b8c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:46.823164 8567 ts_tablet_manager.cc:1905] T cde7715a4b6a4911a2cc7d10b53a77c8 P 479cdfbbfa0a496abd505220830d412e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:46.829319 8566 ts_tablet_manager.cc:1918] T cde7715a4b6a4911a2cc7d10b53a77c8 P 6951c64acb48467bb84a75d1b715b8c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:44:46.829445 8566 log.cc:1196] T cde7715a4b6a4911a2cc7d10b53a77c8 P 6951c64acb48467bb84a75d1b715b8c1: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-2/wal/wals/cde7715a4b6a4911a2cc7d10b53a77c8
I20260928 08:44:46.829732 8565 ts_tablet_manager.cc:1918] T cde7715a4b6a4911a2cc7d10b53a77c8 P e4860201f08e45c98eea6befb1984108: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:44:46.829895 8566 ts_tablet_manager.cc:1939] T cde7715a4b6a4911a2cc7d10b53a77c8 P 6951c64acb48467bb84a75d1b715b8c1: Deleting consensus metadata
I20260928 08:44:46.829936 8565 log.cc:1196] T cde7715a4b6a4911a2cc7d10b53a77c8 P e4860201f08e45c98eea6befb1984108: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-0/wal/wals/cde7715a4b6a4911a2cc7d10b53a77c8
I20260928 08:44:46.830523 8565 ts_tablet_manager.cc:1939] T cde7715a4b6a4911a2cc7d10b53a77c8 P e4860201f08e45c98eea6befb1984108: Deleting consensus metadata
I20260928 08:44:46.830989 7972 catalog_manager.cc:4841] TS 6951c64acb48467bb84a75d1b715b8c1 (127.0.97.3:34497): tablet cde7715a4b6a4911a2cc7d10b53a77c8 (table test-table [id=fd3371f2879848a08465ff342ae037e7]) successfully deleted
I20260928 08:44:46.831483 8567 ts_tablet_manager.cc:1918] T cde7715a4b6a4911a2cc7d10b53a77c8 P 479cdfbbfa0a496abd505220830d412e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:44:46.831620 8567 log.cc:1196] T cde7715a4b6a4911a2cc7d10b53a77c8 P 479cdfbbfa0a496abd505220830d412e: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-1/wal/wals/cde7715a4b6a4911a2cc7d10b53a77c8
I20260928 08:44:46.831875 7969 catalog_manager.cc:4841] TS e4860201f08e45c98eea6befb1984108 (127.0.97.1:40955): tablet cde7715a4b6a4911a2cc7d10b53a77c8 (table test-table [id=fd3371f2879848a08465ff342ae037e7]) successfully deleted
I20260928 08:44:46.832165 8567 ts_tablet_manager.cc:1939] T cde7715a4b6a4911a2cc7d10b53a77c8 P 479cdfbbfa0a496abd505220830d412e: Deleting consensus metadata
I20260928 08:44:46.833290 7969 catalog_manager.cc:4841] TS 479cdfbbfa0a496abd505220830d412e (127.0.97.2:40415): tablet cde7715a4b6a4911a2cc7d10b53a77c8 (table test-table [id=fd3371f2879848a08465ff342ae037e7]) successfully deleted
Sep 28 08:44:46 dist-test-slave-n6xs krb5kdc[7940](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 08:44:46 dist-test-slave-n6xs krb5kdc[7940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:44:46 dist-test-slave-n6xs krb5kdc[7940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 08:44:46.877671 8574 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:44:46.864129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35540 (local address 127.0.97.62:46411)
0928 08:44:46.864663 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:46.864671 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:46.864949 (+ 278us) server_negotiation.cc:407] Connection header received
0928 08:44:46.865172 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:46.865177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:46.865250 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:46.865361 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:46.866654 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:46.867399 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:46.868673 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:46.869075 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:46.869212 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:46.877327 (+ 8115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35540: BlockingRecv error: recv got EOF from 127.0.0.1:35540 (error 108)
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":264,"threads_started":1}
Sep 28 08:44:46 dist-test-slave-n6xs krb5kdc[7940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM
I20260928 08:44:46.894670 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:46.880435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35550 (local address 127.0.97.62:46411)
0928 08:44:46.880600 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:46.880605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:46.880934 (+ 329us) server_negotiation.cc:407] Connection header received
0928 08:44:46.881038 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:46.881043 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:46.881123 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:46.881231 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:46.882266 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:46.883340 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:46.884729 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:46.885348 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:46.885461 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:46.889245 (+ 3784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:46.889249 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:46.889253 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:46.889281 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:46.891986 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:46.892753 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:46.892757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:46.892760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:46.892873 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:46.893363 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:46.893366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:46.893368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:46.893646 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:46.893835 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:46.894232 (+ 397us) server_negotiation.cc:299] Negotiation successful
0928 08:44:46.894433 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 08:44:46.898315 7983 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:44:46.898923 7983 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:44:46.906010 8400 tablet_service.cc:1467] Processing CreateTablet for tablet b7f97f386e1c4629a8c42cda544fae76 (DEFAULT_TABLE table=test-table [id=97e81bcd1eae49179dec4466a960cef2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:46.906080 8261 tablet_service.cc:1467] Processing CreateTablet for tablet b7f97f386e1c4629a8c42cda544fae76 (DEFAULT_TABLE table=test-table [id=97e81bcd1eae49179dec4466a960cef2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:46.906464 8261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7f97f386e1c4629a8c42cda544fae76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:46.906473 8400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7f97f386e1c4629a8c42cda544fae76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:46.908226 8122 tablet_service.cc:1467] Processing CreateTablet for tablet b7f97f386e1c4629a8c42cda544fae76 (DEFAULT_TABLE table=test-table [id=97e81bcd1eae49179dec4466a960cef2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:46.908648 8122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7f97f386e1c4629a8c42cda544fae76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:46.920396 8582 tablet_bootstrap.cc:492] T b7f97f386e1c4629a8c42cda544fae76 P e4860201f08e45c98eea6befb1984108: Bootstrap starting.
I20260928 08:44:46.922514 8582 tablet_bootstrap.cc:654] T b7f97f386e1c4629a8c42cda544fae76 P e4860201f08e45c98eea6befb1984108: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:46.924826 8581 tablet_bootstrap.cc:492] T b7f97f386e1c4629a8c42cda544fae76 P 6951c64acb48467bb84a75d1b715b8c1: Bootstrap starting.
I20260928 08:44:46.925151 8583 tablet_bootstrap.cc:492] T b7f97f386e1c4629a8c42cda544fae76 P 479cdfbbfa0a496abd505220830d412e: Bootstrap starting.
I20260928 08:44:46.926433 8582 tablet_bootstrap.cc:492] T b7f97f386e1c4629a8c42cda544fae76 P e4860201f08e45c98eea6befb1984108: No bootstrap required, opened a new log
I20260928 08:44:46.926574 8582 ts_tablet_manager.cc:1397] T b7f97f386e1c4629a8c42cda544fae76 P e4860201f08e45c98eea6befb1984108: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 08:44:46.926934 8583 tablet_bootstrap.cc:654] T b7f97f386e1c4629a8c42cda544fae76 P 479cdfbbfa0a496abd505220830d412e: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:46.927240 8581 tablet_bootstrap.cc:654] T b7f97f386e1c4629a8c42cda544fae76 P 6951c64acb48467bb84a75d1b715b8c1: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:46.927462 8582 raft_consensus.cc:348] T b7f97f386e1c4629a8c42cda544fae76 P e4860201f08e45c98eea6befb1984108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } }
I20260928 08:44:46.927748 8582 raft_consensus.cc:374] T b7f97f386e1c4629a8c42cda544fae76 P e4860201f08e45c98eea6befb1984108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:46.927878 8582 raft_consensus.cc:729] T b7f97f386e1c4629a8c42cda544fae76 P e4860201f08e45c98eea6befb1984108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4860201f08e45c98eea6befb1984108, State: Initialized, Role: FOLLOWER
I20260928 08:44:46.928210 8582 consensus_queue.cc:260] T b7f97f386e1c4629a8c42cda544fae76 P e4860201f08e45c98eea6befb1984108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } }
I20260928 08:44:46.931706 8582 ts_tablet_manager.cc:1428] T b7f97f386e1c4629a8c42cda544fae76 P e4860201f08e45c98eea6befb1984108: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260928 08:44:46.935250 8583 tablet_bootstrap.cc:492] T b7f97f386e1c4629a8c42cda544fae76 P 479cdfbbfa0a496abd505220830d412e: No bootstrap required, opened a new log
I20260928 08:44:46.935461 8583 ts_tablet_manager.cc:1397] T b7f97f386e1c4629a8c42cda544fae76 P 479cdfbbfa0a496abd505220830d412e: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260928 08:44:46.936172 8583 raft_consensus.cc:348] T b7f97f386e1c4629a8c42cda544fae76 P 479cdfbbfa0a496abd505220830d412e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } }
I20260928 08:44:46.936426 8583 raft_consensus.cc:374] T b7f97f386e1c4629a8c42cda544fae76 P 479cdfbbfa0a496abd505220830d412e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:46.936523 8583 raft_consensus.cc:729] T b7f97f386e1c4629a8c42cda544fae76 P 479cdfbbfa0a496abd505220830d412e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 479cdfbbfa0a496abd505220830d412e, State: Initialized, Role: FOLLOWER
I20260928 08:44:46.936798 8583 consensus_queue.cc:260] T b7f97f386e1c4629a8c42cda544fae76 P 479cdfbbfa0a496abd505220830d412e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } }
I20260928 08:44:46.937340 8583 ts_tablet_manager.cc:1428] T b7f97f386e1c4629a8c42cda544fae76 P 479cdfbbfa0a496abd505220830d412e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 08:44:46.938956 8581 tablet_bootstrap.cc:492] T b7f97f386e1c4629a8c42cda544fae76 P 6951c64acb48467bb84a75d1b715b8c1: No bootstrap required, opened a new log
I20260928 08:44:46.939114 8581 ts_tablet_manager.cc:1397] T b7f97f386e1c4629a8c42cda544fae76 P 6951c64acb48467bb84a75d1b715b8c1: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20260928 08:44:46.940066 8581 raft_consensus.cc:348] T b7f97f386e1c4629a8c42cda544fae76 P 6951c64acb48467bb84a75d1b715b8c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } }
I20260928 08:44:46.940311 8581 raft_consensus.cc:374] T b7f97f386e1c4629a8c42cda544fae76 P 6951c64acb48467bb84a75d1b715b8c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:46.940423 8581 raft_consensus.cc:729] T b7f97f386e1c4629a8c42cda544fae76 P 6951c64acb48467bb84a75d1b715b8c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6951c64acb48467bb84a75d1b715b8c1, State: Initialized, Role: FOLLOWER
I20260928 08:44:46.940743 8581 consensus_queue.cc:260] T b7f97f386e1c4629a8c42cda544fae76 P 6951c64acb48467bb84a75d1b715b8c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } }
I20260928 08:44:46.941368 8581 ts_tablet_manager.cc:1428] T b7f97f386e1c4629a8c42cda544fae76 P 6951c64acb48467bb84a75d1b715b8c1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260928 08:44:46.949023 8563 raft_consensus.cc:482] T b7f97f386e1c4629a8c42cda544fae76 P 479cdfbbfa0a496abd505220830d412e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:44:46.949254 8563 raft_consensus.cc:504] T b7f97f386e1c4629a8c42cda544fae76 P 479cdfbbfa0a496abd505220830d412e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } }
I20260928 08:44:46.949963 8563 leader_election.cc:288] T b7f97f386e1c4629a8c42cda544fae76 P 479cdfbbfa0a496abd505220830d412e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4860201f08e45c98eea6befb1984108 (127.0.97.1:40955), 6951c64acb48467bb84a75d1b715b8c1 (127.0.97.3:34497)
I20260928 08:44:46.950525 8141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7f97f386e1c4629a8c42cda544fae76" candidate_uuid: "479cdfbbfa0a496abd505220830d412e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4860201f08e45c98eea6befb1984108" is_pre_election: true
I20260928 08:44:46.950819 8141 raft_consensus.cc:2454] T b7f97f386e1c4629a8c42cda544fae76 P e4860201f08e45c98eea6befb1984108 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 479cdfbbfa0a496abd505220830d412e in term 0.
I20260928 08:44:46.950907 8424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7f97f386e1c4629a8c42cda544fae76" candidate_uuid: "479cdfbbfa0a496abd505220830d412e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6951c64acb48467bb84a75d1b715b8c1" is_pre_election: true
I20260928 08:44:46.951157 8424 raft_consensus.cc:2454] T b7f97f386e1c4629a8c42cda544fae76 P 6951c64acb48467bb84a75d1b715b8c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 479cdfbbfa0a496abd505220830d412e in term 0.
I20260928 08:44:46.951289 8212 leader_election.cc:302] T b7f97f386e1c4629a8c42cda544fae76 P 479cdfbbfa0a496abd505220830d412e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 479cdfbbfa0a496abd505220830d412e, e4860201f08e45c98eea6befb1984108; no voters:
I20260928 08:44:46.951694 8563 raft_consensus.cc:2781] T b7f97f386e1c4629a8c42cda544fae76 P 479cdfbbfa0a496abd505220830d412e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:44:46.951825 8563 raft_consensus.cc:482] T b7f97f386e1c4629a8c42cda544fae76 P 479cdfbbfa0a496abd505220830d412e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:44:46.951915 8563 raft_consensus.cc:3037] T b7f97f386e1c4629a8c42cda544fae76 P 479cdfbbfa0a496abd505220830d412e [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:46.953935 8563 raft_consensus.cc:504] T b7f97f386e1c4629a8c42cda544fae76 P 479cdfbbfa0a496abd505220830d412e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } }
I20260928 08:44:46.954489 8563 leader_election.cc:288] T b7f97f386e1c4629a8c42cda544fae76 P 479cdfbbfa0a496abd505220830d412e [CANDIDATE]: Term 1 election: Requested vote from peers e4860201f08e45c98eea6befb1984108 (127.0.97.1:40955), 6951c64acb48467bb84a75d1b715b8c1 (127.0.97.3:34497)
I20260928 08:44:46.954873 8141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7f97f386e1c4629a8c42cda544fae76" candidate_uuid: "479cdfbbfa0a496abd505220830d412e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4860201f08e45c98eea6befb1984108"
I20260928 08:44:46.954962 8424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7f97f386e1c4629a8c42cda544fae76" candidate_uuid: "479cdfbbfa0a496abd505220830d412e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6951c64acb48467bb84a75d1b715b8c1"
I20260928 08:44:46.955071 8141 raft_consensus.cc:3037] T b7f97f386e1c4629a8c42cda544fae76 P e4860201f08e45c98eea6befb1984108 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:46.955204 8424 raft_consensus.cc:3037] T b7f97f386e1c4629a8c42cda544fae76 P 6951c64acb48467bb84a75d1b715b8c1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:46.957439 8424 raft_consensus.cc:2454] T b7f97f386e1c4629a8c42cda544fae76 P 6951c64acb48467bb84a75d1b715b8c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 479cdfbbfa0a496abd505220830d412e in term 1.
I20260928 08:44:46.957958 8215 leader_election.cc:302] T b7f97f386e1c4629a8c42cda544fae76 P 479cdfbbfa0a496abd505220830d412e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 479cdfbbfa0a496abd505220830d412e, 6951c64acb48467bb84a75d1b715b8c1; no voters:
I20260928 08:44:46.958074 8141 raft_consensus.cc:2454] T b7f97f386e1c4629a8c42cda544fae76 P e4860201f08e45c98eea6befb1984108 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 479cdfbbfa0a496abd505220830d412e in term 1.
I20260928 08:44:46.958314 8563 raft_consensus.cc:2781] T b7f97f386e1c4629a8c42cda544fae76 P 479cdfbbfa0a496abd505220830d412e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:44:46.958515 8563 raft_consensus.cc:686] T b7f97f386e1c4629a8c42cda544fae76 P 479cdfbbfa0a496abd505220830d412e [term 1 LEADER]: Becoming Leader. State: Replica: 479cdfbbfa0a496abd505220830d412e, State: Running, Role: LEADER
I20260928 08:44:46.959299 8563 consensus_queue.cc:237] T b7f97f386e1c4629a8c42cda544fae76 P 479cdfbbfa0a496abd505220830d412e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } }
I20260928 08:44:46.962312 7985 catalog_manager.cc:5495] T b7f97f386e1c4629a8c42cda544fae76 P 479cdfbbfa0a496abd505220830d412e reported cstate change: term changed from 0 to 1, leader changed from <none> to 479cdfbbfa0a496abd505220830d412e (127.0.97.2). New cstate: current_term: 1 leader_uuid: "479cdfbbfa0a496abd505220830d412e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } health_report { overall_health: UNKNOWN } } }
I20260928 08:44:46.986178 8537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:46.980490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53284 (local address 127.0.97.2:40415)
0928 08:44:46.980673 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:46.980677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:46.980743 (+ 66us) server_negotiation.cc:407] Connection header received
0928 08:44:46.980965 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:46.980968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:46.981026 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:46.981095 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:44:46.981612 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:46.982361 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:46.984361 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:46.984778 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:46.984875 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:46.985345 (+ 470us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:44:46.985448 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:44:46.985517 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:46.985720 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 08:44:46.985790 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 08:44:46.988996 8141 raft_consensus.cc:1261] T b7f97f386e1c4629a8c42cda544fae76 P e4860201f08e45c98eea6befb1984108 [term 1 FOLLOWER]: Refusing update from remote peer 479cdfbbfa0a496abd505220830d412e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:44:46.989112 8424 raft_consensus.cc:1261] T b7f97f386e1c4629a8c42cda544fae76 P 6951c64acb48467bb84a75d1b715b8c1 [term 1 FOLLOWER]: Refusing update from remote peer 479cdfbbfa0a496abd505220830d412e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:44:46.989701 8563 consensus_queue.cc:1035] T b7f97f386e1c4629a8c42cda544fae76 P 479cdfbbfa0a496abd505220830d412e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:44:46.989995 8515 consensus_queue.cc:1035] T b7f97f386e1c4629a8c42cda544fae76 P 479cdfbbfa0a496abd505220830d412e [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:44:47.004598 7985 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:44:47.004858 7985 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:44:47.008122 7985 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef140c385b0b49229ff2e2ddd2a91894: Sending DeleteTablet for 3 replicas of tablet b7f97f386e1c4629a8c42cda544fae76
I20260928 08:44:47.008836 8122 tablet_service.cc:1514] Processing DeleteTablet for tablet b7f97f386e1c4629a8c42cda544fae76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:47 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46096
I20260928 08:44:47.009130 8261 tablet_service.cc:1514] Processing DeleteTablet for tablet b7f97f386e1c4629a8c42cda544fae76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:47 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53268
I20260928 08:44:47.009393 8565 tablet_replica.cc:330] T b7f97f386e1c4629a8c42cda544fae76 P e4860201f08e45c98eea6befb1984108: stopping tablet replica
I20260928 08:44:47.009585 8565 raft_consensus.cc:2229] T b7f97f386e1c4629a8c42cda544fae76 P e4860201f08e45c98eea6befb1984108 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:44:47.009677 8567 tablet_replica.cc:330] T b7f97f386e1c4629a8c42cda544fae76 P 479cdfbbfa0a496abd505220830d412e: stopping tablet replica
I20260928 08:44:47.009822 8565 raft_consensus.cc:2258] T b7f97f386e1c4629a8c42cda544fae76 P e4860201f08e45c98eea6befb1984108 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:47.009891 8567 raft_consensus.cc:2229] T b7f97f386e1c4629a8c42cda544fae76 P 479cdfbbfa0a496abd505220830d412e [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:44:47.010227 8567 raft_consensus.cc:2258] T b7f97f386e1c4629a8c42cda544fae76 P 479cdfbbfa0a496abd505220830d412e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:47.010807 8565 ts_tablet_manager.cc:1905] T b7f97f386e1c4629a8c42cda544fae76 P e4860201f08e45c98eea6befb1984108: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:47.010972 8400 tablet_service.cc:1514] Processing DeleteTablet for tablet b7f97f386e1c4629a8c42cda544fae76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:47 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37942
I20260928 08:44:47.011271 8566 tablet_replica.cc:330] T b7f97f386e1c4629a8c42cda544fae76 P 6951c64acb48467bb84a75d1b715b8c1: stopping tablet replica
I20260928 08:44:47.011518 8566 raft_consensus.cc:2229] T b7f97f386e1c4629a8c42cda544fae76 P 6951c64acb48467bb84a75d1b715b8c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:44:47.011731 8566 raft_consensus.cc:2258] T b7f97f386e1c4629a8c42cda544fae76 P 6951c64acb48467bb84a75d1b715b8c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:47.012883 8566 ts_tablet_manager.cc:1905] T b7f97f386e1c4629a8c42cda544fae76 P 6951c64acb48467bb84a75d1b715b8c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:47.017457 8565 ts_tablet_manager.cc:1918] T b7f97f386e1c4629a8c42cda544fae76 P e4860201f08e45c98eea6befb1984108: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:44:47.017611 8565 log.cc:1196] T b7f97f386e1c4629a8c42cda544fae76 P e4860201f08e45c98eea6befb1984108: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-0/wal/wals/b7f97f386e1c4629a8c42cda544fae76
I20260928 08:44:47.018014 8565 ts_tablet_manager.cc:1939] T b7f97f386e1c4629a8c42cda544fae76 P e4860201f08e45c98eea6befb1984108: Deleting consensus metadata
I20260928 08:44:47.018972 7969 catalog_manager.cc:4841] TS e4860201f08e45c98eea6befb1984108 (127.0.97.1:40955): tablet b7f97f386e1c4629a8c42cda544fae76 (table test-table [id=97e81bcd1eae49179dec4466a960cef2]) successfully deleted
I20260928 08:44:47.020521 8567 ts_tablet_manager.cc:1905] T b7f97f386e1c4629a8c42cda544fae76 P 479cdfbbfa0a496abd505220830d412e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:47.024230 8566 ts_tablet_manager.cc:1918] T b7f97f386e1c4629a8c42cda544fae76 P 6951c64acb48467bb84a75d1b715b8c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:44:47.024397 8566 log.cc:1196] T b7f97f386e1c4629a8c42cda544fae76 P 6951c64acb48467bb84a75d1b715b8c1: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-2/wal/wals/b7f97f386e1c4629a8c42cda544fae76
I20260928 08:44:47.010249 7985 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 08:44:47.024855 8566 ts_tablet_manager.cc:1939] T b7f97f386e1c4629a8c42cda544fae76 P 6951c64acb48467bb84a75d1b715b8c1: Deleting consensus metadata
W20260928 08:44:47.025178 7985 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:44:47.025822 7972 catalog_manager.cc:4841] TS 6951c64acb48467bb84a75d1b715b8c1 (127.0.97.3:34497): tablet b7f97f386e1c4629a8c42cda544fae76 (table test-table [id=97e81bcd1eae49179dec4466a960cef2]) successfully deleted
I20260928 08:44:47.026939 8567 ts_tablet_manager.cc:1918] T b7f97f386e1c4629a8c42cda544fae76 P 479cdfbbfa0a496abd505220830d412e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:44:47.027117 8567 log.cc:1196] T b7f97f386e1c4629a8c42cda544fae76 P 479cdfbbfa0a496abd505220830d412e: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-1/wal/wals/b7f97f386e1c4629a8c42cda544fae76
I20260928 08:44:47.027570 8567 ts_tablet_manager.cc:1939] T b7f97f386e1c4629a8c42cda544fae76 P 479cdfbbfa0a496abd505220830d412e: Deleting consensus metadata
I20260928 08:44:47.028710 7969 catalog_manager.cc:4841] TS 479cdfbbfa0a496abd505220830d412e (127.0.97.2:40415): tablet b7f97f386e1c4629a8c42cda544fae76 (table test-table [id=97e81bcd1eae49179dec4466a960cef2]) successfully deleted
I20260928 08:44:47.034086 8122 tablet_service.cc:1467] Processing CreateTablet for tablet a67fe85812a84af5a28d26b96b49cd15 (DEFAULT_TABLE table=test-table [id=ef9e07fadd214a4bbb2f917e6a9bdbbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:47.034536 8261 tablet_service.cc:1467] Processing CreateTablet for tablet a67fe85812a84af5a28d26b96b49cd15 (DEFAULT_TABLE table=test-table [id=ef9e07fadd214a4bbb2f917e6a9bdbbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:47.034968 8261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a67fe85812a84af5a28d26b96b49cd15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:47.036263 8400 tablet_service.cc:1467] Processing CreateTablet for tablet a67fe85812a84af5a28d26b96b49cd15 (DEFAULT_TABLE table=test-table [id=ef9e07fadd214a4bbb2f917e6a9bdbbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:44:47.036790 8400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a67fe85812a84af5a28d26b96b49cd15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:47.038775 8122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a67fe85812a84af5a28d26b96b49cd15. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:47.045753 8582 tablet_bootstrap.cc:492] T a67fe85812a84af5a28d26b96b49cd15 P e4860201f08e45c98eea6befb1984108: Bootstrap starting.
I20260928 08:44:47.047566 8582 tablet_bootstrap.cc:654] T a67fe85812a84af5a28d26b96b49cd15 P e4860201f08e45c98eea6befb1984108: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:47.049211 8583 tablet_bootstrap.cc:492] T a67fe85812a84af5a28d26b96b49cd15 P 479cdfbbfa0a496abd505220830d412e: Bootstrap starting.
I20260928 08:44:47.049405 8581 tablet_bootstrap.cc:492] T a67fe85812a84af5a28d26b96b49cd15 P 6951c64acb48467bb84a75d1b715b8c1: Bootstrap starting.
I20260928 08:44:47.050644 8583 tablet_bootstrap.cc:654] T a67fe85812a84af5a28d26b96b49cd15 P 479cdfbbfa0a496abd505220830d412e: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:47.050834 8581 tablet_bootstrap.cc:654] T a67fe85812a84af5a28d26b96b49cd15 P 6951c64acb48467bb84a75d1b715b8c1: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:47.052040 8582 tablet_bootstrap.cc:492] T a67fe85812a84af5a28d26b96b49cd15 P e4860201f08e45c98eea6befb1984108: No bootstrap required, opened a new log
I20260928 08:44:47.052206 8582 ts_tablet_manager.cc:1397] T a67fe85812a84af5a28d26b96b49cd15 P e4860201f08e45c98eea6befb1984108: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20260928 08:44:47.052923 8582 raft_consensus.cc:348] T a67fe85812a84af5a28d26b96b49cd15 P e4860201f08e45c98eea6befb1984108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } }
I20260928 08:44:47.053139 8581 tablet_bootstrap.cc:492] T a67fe85812a84af5a28d26b96b49cd15 P 6951c64acb48467bb84a75d1b715b8c1: No bootstrap required, opened a new log
I20260928 08:44:47.053201 8582 raft_consensus.cc:374] T a67fe85812a84af5a28d26b96b49cd15 P e4860201f08e45c98eea6befb1984108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:47.053278 8582 raft_consensus.cc:729] T a67fe85812a84af5a28d26b96b49cd15 P e4860201f08e45c98eea6befb1984108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4860201f08e45c98eea6befb1984108, State: Initialized, Role: FOLLOWER
I20260928 08:44:47.053385 8581 ts_tablet_manager.cc:1397] T a67fe85812a84af5a28d26b96b49cd15 P 6951c64acb48467bb84a75d1b715b8c1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260928 08:44:47.053540 8582 consensus_queue.cc:260] T a67fe85812a84af5a28d26b96b49cd15 P e4860201f08e45c98eea6befb1984108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } }
I20260928 08:44:47.054425 8581 raft_consensus.cc:348] T a67fe85812a84af5a28d26b96b49cd15 P 6951c64acb48467bb84a75d1b715b8c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } }
I20260928 08:44:47.054718 8581 raft_consensus.cc:374] T a67fe85812a84af5a28d26b96b49cd15 P 6951c64acb48467bb84a75d1b715b8c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:47.054816 8581 raft_consensus.cc:729] T a67fe85812a84af5a28d26b96b49cd15 P 6951c64acb48467bb84a75d1b715b8c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6951c64acb48467bb84a75d1b715b8c1, State: Initialized, Role: FOLLOWER
I20260928 08:44:47.054817 8583 tablet_bootstrap.cc:492] T a67fe85812a84af5a28d26b96b49cd15 P 479cdfbbfa0a496abd505220830d412e: No bootstrap required, opened a new log
I20260928 08:44:47.055182 8583 ts_tablet_manager.cc:1397] T a67fe85812a84af5a28d26b96b49cd15 P 479cdfbbfa0a496abd505220830d412e: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260928 08:44:47.055124 8581 consensus_queue.cc:260] T a67fe85812a84af5a28d26b96b49cd15 P 6951c64acb48467bb84a75d1b715b8c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } }
I20260928 08:44:47.055763 8581 ts_tablet_manager.cc:1428] T a67fe85812a84af5a28d26b96b49cd15 P 6951c64acb48467bb84a75d1b715b8c1: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260928 08:44:47.056329 8582 ts_tablet_manager.cc:1428] T a67fe85812a84af5a28d26b96b49cd15 P e4860201f08e45c98eea6befb1984108: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260928 08:44:47.056210 8583 raft_consensus.cc:348] T a67fe85812a84af5a28d26b96b49cd15 P 479cdfbbfa0a496abd505220830d412e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } }
I20260928 08:44:47.056600 8583 raft_consensus.cc:374] T a67fe85812a84af5a28d26b96b49cd15 P 479cdfbbfa0a496abd505220830d412e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:47.056739 8583 raft_consensus.cc:729] T a67fe85812a84af5a28d26b96b49cd15 P 479cdfbbfa0a496abd505220830d412e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 479cdfbbfa0a496abd505220830d412e, State: Initialized, Role: FOLLOWER
I20260928 08:44:47.057070 8583 consensus_queue.cc:260] T a67fe85812a84af5a28d26b96b49cd15 P 479cdfbbfa0a496abd505220830d412e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } }
I20260928 08:44:47.058722 8583 ts_tablet_manager.cc:1428] T a67fe85812a84af5a28d26b96b49cd15 P 479cdfbbfa0a496abd505220830d412e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 08:44:47.173008 8501 raft_consensus.cc:482] T a67fe85812a84af5a28d26b96b49cd15 P e4860201f08e45c98eea6befb1984108 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:44:47.173308 8501 raft_consensus.cc:504] T a67fe85812a84af5a28d26b96b49cd15 P e4860201f08e45c98eea6befb1984108 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } }
I20260928 08:44:47.175118 8501 leader_election.cc:288] T a67fe85812a84af5a28d26b96b49cd15 P e4860201f08e45c98eea6befb1984108 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 479cdfbbfa0a496abd505220830d412e (127.0.97.2:40415), 6951c64acb48467bb84a75d1b715b8c1 (127.0.97.3:34497)
I20260928 08:44:47.183765 8545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:47.175542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48211 (local address 127.0.97.3:34497)
0928 08:44:47.175710 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:47.175717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:47.176643 (+ 926us) server_negotiation.cc:407] Connection header received
0928 08:44:47.176877 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:47.176882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:47.176952 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:47.177042 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:47.177840 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:47.178861 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:47.181585 (+ 2724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:47.182838 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:47.182985 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:47.183089 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:47.183414 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 08:44:47.183579 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 08:44:47.184389 8594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:47.175451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34497 (local address 127.0.97.1:48211)
0928 08:44:47.176524 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:47.176542 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:47.176720 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:47.177204 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:47.177208 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:47.177237 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:47.177670 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:47.177680 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:47.179031 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:47.179036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:47.181446 (+ 2410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:47.181456 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:47.183041 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:47.183050 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:47.183224 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:47.183260 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 08:44:47.183993 (+ 733us) client_negotiation.cc:240] Negotiation successful
0928 08:44:47.184122 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":945,"mutex_wait_us":160,"thread_start_us":427,"threads_started":1}
I20260928 08:44:47.184996 8424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a67fe85812a84af5a28d26b96b49cd15" candidate_uuid: "e4860201f08e45c98eea6befb1984108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6951c64acb48467bb84a75d1b715b8c1" is_pre_election: true
I20260928 08:44:47.185245 8424 raft_consensus.cc:2454] T a67fe85812a84af5a28d26b96b49cd15 P 6951c64acb48467bb84a75d1b715b8c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4860201f08e45c98eea6befb1984108 in term 0.
I20260928 08:44:47.185747 8537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:47.176563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60271 (local address 127.0.97.2:40415)
0928 08:44:47.176737 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:47.176742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:47.176766 (+ 24us) server_negotiation.cc:407] Connection header received
0928 08:44:47.176842 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:47.176848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:47.176908 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:47.177016 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:44:47.177690 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:47.178870 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:47.183186 (+ 4316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:47.184547 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:47.184665 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:47.184768 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:47.185350 (+ 582us) server_negotiation.cc:299] Negotiation successful
0928 08:44:47.185492 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 08:44:47.185688 8075 leader_election.cc:302] T a67fe85812a84af5a28d26b96b49cd15 P e4860201f08e45c98eea6befb1984108 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6951c64acb48467bb84a75d1b715b8c1, e4860201f08e45c98eea6befb1984108; no voters:
I20260928 08:44:47.185959 8595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:47.175421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40415 (local address 127.0.97.1:60271)
0928 08:44:47.176144 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:47.176164 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:47.176313 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:47.177032 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:47.177037 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:47.177061 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:44:47.177423 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:47.177469 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:47.179031 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:47.179036 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:47.181421 (+ 2385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:47.181439 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:47.184953 (+ 3514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:47.184959 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:47.185134 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:47.185170 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 08:44:47.185310 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 08:44:47.185556 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":80,"thread_start_us":799,"threads_started":1}
I20260928 08:44:47.186187 8501 raft_consensus.cc:2781] T a67fe85812a84af5a28d26b96b49cd15 P e4860201f08e45c98eea6befb1984108 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:44:47.186290 8501 raft_consensus.cc:482] T a67fe85812a84af5a28d26b96b49cd15 P e4860201f08e45c98eea6befb1984108 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:44:47.186368 8501 raft_consensus.cc:3037] T a67fe85812a84af5a28d26b96b49cd15 P e4860201f08e45c98eea6befb1984108 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:47.189321 8501 raft_consensus.cc:504] T a67fe85812a84af5a28d26b96b49cd15 P e4860201f08e45c98eea6befb1984108 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } }
I20260928 08:44:47.189713 8281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a67fe85812a84af5a28d26b96b49cd15" candidate_uuid: "e4860201f08e45c98eea6befb1984108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "479cdfbbfa0a496abd505220830d412e" is_pre_election: true
I20260928 08:44:47.189996 8281 raft_consensus.cc:2454] T a67fe85812a84af5a28d26b96b49cd15 P 479cdfbbfa0a496abd505220830d412e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4860201f08e45c98eea6befb1984108 in term 0.
I20260928 08:44:47.190346 8424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a67fe85812a84af5a28d26b96b49cd15" candidate_uuid: "e4860201f08e45c98eea6befb1984108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6951c64acb48467bb84a75d1b715b8c1"
I20260928 08:44:47.190366 8281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a67fe85812a84af5a28d26b96b49cd15" candidate_uuid: "e4860201f08e45c98eea6befb1984108" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "479cdfbbfa0a496abd505220830d412e"
I20260928 08:44:47.190526 8424 raft_consensus.cc:3037] T a67fe85812a84af5a28d26b96b49cd15 P 6951c64acb48467bb84a75d1b715b8c1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:47.190533 8281 raft_consensus.cc:3037] T a67fe85812a84af5a28d26b96b49cd15 P 479cdfbbfa0a496abd505220830d412e [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:47.192564 8501 leader_election.cc:288] T a67fe85812a84af5a28d26b96b49cd15 P e4860201f08e45c98eea6befb1984108 [CANDIDATE]: Term 1 election: Requested vote from peers 479cdfbbfa0a496abd505220830d412e (127.0.97.2:40415), 6951c64acb48467bb84a75d1b715b8c1 (127.0.97.3:34497)
I20260928 08:44:47.192705 8281 raft_consensus.cc:2454] T a67fe85812a84af5a28d26b96b49cd15 P 479cdfbbfa0a496abd505220830d412e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4860201f08e45c98eea6befb1984108 in term 1.
I20260928 08:44:47.193174 8072 leader_election.cc:302] T a67fe85812a84af5a28d26b96b49cd15 P e4860201f08e45c98eea6befb1984108 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 479cdfbbfa0a496abd505220830d412e, e4860201f08e45c98eea6befb1984108; no voters:
I20260928 08:44:47.193374 8424 raft_consensus.cc:2454] T a67fe85812a84af5a28d26b96b49cd15 P 6951c64acb48467bb84a75d1b715b8c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4860201f08e45c98eea6befb1984108 in term 1.
I20260928 08:44:47.193480 8501 raft_consensus.cc:2781] T a67fe85812a84af5a28d26b96b49cd15 P e4860201f08e45c98eea6befb1984108 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:44:47.194075 8501 raft_consensus.cc:686] T a67fe85812a84af5a28d26b96b49cd15 P e4860201f08e45c98eea6befb1984108 [term 1 LEADER]: Becoming Leader. State: Replica: e4860201f08e45c98eea6befb1984108, State: Running, Role: LEADER
I20260928 08:44:47.194542 8501 consensus_queue.cc:237] T a67fe85812a84af5a28d26b96b49cd15 P e4860201f08e45c98eea6befb1984108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } }
I20260928 08:44:47.198149 7984 catalog_manager.cc:5495] T a67fe85812a84af5a28d26b96b49cd15 P e4860201f08e45c98eea6befb1984108 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4860201f08e45c98eea6befb1984108 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "e4860201f08e45c98eea6befb1984108" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4860201f08e45c98eea6befb1984108" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 } health_report { overall_health: UNKNOWN } } }
I20260928 08:44:47.255929 8598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:47.247610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46098 (local address 127.0.97.1:40955)
0928 08:44:47.248292 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:47.248298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:47.248320 (+ 22us) server_negotiation.cc:407] Connection header received
0928 08:44:47.248417 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:47.248422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:47.248543 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:47.248732 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:44:47.249167 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:47.250300 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:47.252536 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:47.253277 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:47.253398 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:47.254186 (+ 788us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:44:47.255119 (+ 933us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:44:47.255302 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:47.255386 (+ 84us) server_negotiation.cc:299] Negotiation successful
0928 08:44:47.255471 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":399,"threads_started":1}
I20260928 08:44:47.274331 8545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:47.268085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34901 (local address 127.0.97.3:34497)
0928 08:44:47.268281 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:47.268289 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:47.268535 (+ 246us) server_negotiation.cc:407] Connection header received
0928 08:44:47.268792 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:47.268799 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:47.268896 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:47.269002 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:44:47.269636 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:47.270863 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:47.272745 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:47.273161 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:47.273257 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:47.273694 (+ 437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:44:47.273855 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:44:47.273930 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:47.274067 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 08:44:47.274147 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 08:44:47.274398 8601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:47.267788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34497 (local address 127.0.97.1:34901)
0928 08:44:47.268438 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:47.268453 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:47.268617 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:47.269053 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:47.269058 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:47.269071 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:44:47.269481 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:47.269492 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:47.270997 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:47.271001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:47.272612 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:47.272625 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:47.273306 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:47.273310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:47.273416 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:47.273503 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:44:47.273951 (+ 448us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:44:47.273959 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 08:44:47.274035 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 08:44:47.274133 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":298,"threads_started":1}
I20260928 08:44:47.283177 8281 raft_consensus.cc:1261] T a67fe85812a84af5a28d26b96b49cd15 P 479cdfbbfa0a496abd505220830d412e [term 1 FOLLOWER]: Refusing update from remote peer e4860201f08e45c98eea6befb1984108: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:44:47.283170 8424 raft_consensus.cc:1261] T a67fe85812a84af5a28d26b96b49cd15 P 6951c64acb48467bb84a75d1b715b8c1 [term 1 FOLLOWER]: Refusing update from remote peer e4860201f08e45c98eea6befb1984108: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:44:47.283829 8596 consensus_queue.cc:1035] T a67fe85812a84af5a28d26b96b49cd15 P e4860201f08e45c98eea6befb1984108 [LEADER]: Connected to new peer: Peer: permanent_uuid: "479cdfbbfa0a496abd505220830d412e" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:44:47.284271 8501 consensus_queue.cc:1035] T a67fe85812a84af5a28d26b96b49cd15 P e4860201f08e45c98eea6befb1984108 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6951c64acb48467bb84a75d1b715b8c1" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:44:47.311630 8598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:47.306341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59223 (local address 127.0.97.1:40955)
0928 08:44:47.306494 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:47.306500 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:47.306687 (+ 187us) server_negotiation.cc:407] Connection header received
0928 08:44:47.306853 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:47.306856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:47.306931 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:47.307018 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:44:47.307464 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:47.308378 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:47.309717 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:47.310213 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:47.310333 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:47.310698 (+ 365us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:44:47.310995 (+ 297us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:44:47.311123 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:47.311307 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 08:44:47.311385 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 08:44:47.311901 8608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:47.306119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40955 (local address 127.0.97.3:59223)
0928 08:44:47.306603 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:47.306620 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:47.306746 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:47.307052 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:47.307056 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:47.307070 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 08:44:47.307290 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:47.307298 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:47.308516 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:47.308522 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:47.309565 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:47.309579 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:47.310343 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:47.310349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:47.310499 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:47.310557 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 08:44:47.311123 (+ 566us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 08:44:47.311145 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 08:44:47.311266 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 08:44:47.311433 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":270,"threads_started":1}
I20260928 08:44:47.350328 7984 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:44:47.350618 7984 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:44:47.353936 7984 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef140c385b0b49229ff2e2ddd2a91894: Sending DeleteTablet for 3 replicas of tablet a67fe85812a84af5a28d26b96b49cd15
I20260928 08:44:47.354722 8122 tablet_service.cc:1514] Processing DeleteTablet for tablet a67fe85812a84af5a28d26b96b49cd15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:47 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46096
I20260928 08:44:47.355022 8261 tablet_service.cc:1514] Processing DeleteTablet for tablet a67fe85812a84af5a28d26b96b49cd15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:47 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:53268
I20260928 08:44:47.355193 8400 tablet_service.cc:1514] Processing DeleteTablet for tablet a67fe85812a84af5a28d26b96b49cd15 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:44:47 UTC) from {username='oryx', principal='oryx/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:37942
I20260928 08:44:47.355376 8565 tablet_replica.cc:330] T a67fe85812a84af5a28d26b96b49cd15 P e4860201f08e45c98eea6befb1984108: stopping tablet replica
I20260928 08:44:47.355569 8566 tablet_replica.cc:330] T a67fe85812a84af5a28d26b96b49cd15 P 6951c64acb48467bb84a75d1b715b8c1: stopping tablet replica
I20260928 08:44:47.355569 8565 raft_consensus.cc:2229] T a67fe85812a84af5a28d26b96b49cd15 P e4860201f08e45c98eea6befb1984108 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:44:47.355785 8566 raft_consensus.cc:2229] T a67fe85812a84af5a28d26b96b49cd15 P 6951c64acb48467bb84a75d1b715b8c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:44:47.356062 8566 raft_consensus.cc:2258] T a67fe85812a84af5a28d26b96b49cd15 P 6951c64acb48467bb84a75d1b715b8c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:47.356123 8565 raft_consensus.cc:2258] T a67fe85812a84af5a28d26b96b49cd15 P e4860201f08e45c98eea6befb1984108 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:47.356495 8567 tablet_replica.cc:330] T a67fe85812a84af5a28d26b96b49cd15 P 479cdfbbfa0a496abd505220830d412e: stopping tablet replica
I20260928 08:44:47.356702 8567 raft_consensus.cc:2229] T a67fe85812a84af5a28d26b96b49cd15 P 479cdfbbfa0a496abd505220830d412e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:44:47.356917 8567 raft_consensus.cc:2258] T a67fe85812a84af5a28d26b96b49cd15 P 479cdfbbfa0a496abd505220830d412e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:44:47.357265 8565 ts_tablet_manager.cc:1905] T a67fe85812a84af5a28d26b96b49cd15 P e4860201f08e45c98eea6befb1984108: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:47.357309 8566 ts_tablet_manager.cc:1905] T a67fe85812a84af5a28d26b96b49cd15 P 6951c64acb48467bb84a75d1b715b8c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:47.358469 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 8056
I20260928 08:44:47.363745 8566 ts_tablet_manager.cc:1918] T a67fe85812a84af5a28d26b96b49cd15 P 6951c64acb48467bb84a75d1b715b8c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:44:47.363910 8566 log.cc:1196] T a67fe85812a84af5a28d26b96b49cd15 P 6951c64acb48467bb84a75d1b715b8c1: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-2/wal/wals/a67fe85812a84af5a28d26b96b49cd15
I20260928 08:44:47.364549 8566 ts_tablet_manager.cc:1939] T a67fe85812a84af5a28d26b96b49cd15 P 6951c64acb48467bb84a75d1b715b8c1: Deleting consensus metadata
I20260928 08:44:47.365617 7972 catalog_manager.cc:4841] TS 6951c64acb48467bb84a75d1b715b8c1 (127.0.97.3:34497): tablet a67fe85812a84af5a28d26b96b49cd15 (table test-table [id=ef9e07fadd214a4bbb2f917e6a9bdbbc]) successfully deleted
I20260928 08:44:47.365682 8565 ts_tablet_manager.cc:1918] T a67fe85812a84af5a28d26b96b49cd15 P e4860201f08e45c98eea6befb1984108: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:44:47.365849 8565 log.cc:1196] T a67fe85812a84af5a28d26b96b49cd15 P e4860201f08e45c98eea6befb1984108: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-0/wal/wals/a67fe85812a84af5a28d26b96b49cd15
I20260928 08:44:47.366400 8565 ts_tablet_manager.cc:1939] T a67fe85812a84af5a28d26b96b49cd15 P e4860201f08e45c98eea6befb1984108: Deleting consensus metadata
I20260928 08:44:47.367759 7969 catalog_manager.cc:4841] TS e4860201f08e45c98eea6befb1984108 (127.0.97.1:40955): tablet a67fe85812a84af5a28d26b96b49cd15 (table test-table [id=ef9e07fadd214a4bbb2f917e6a9bdbbc]) successfully deleted
I20260928 08:44:47.368523 8567 ts_tablet_manager.cc:1905] T a67fe85812a84af5a28d26b96b49cd15 P 479cdfbbfa0a496abd505220830d412e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:44:47.375470 8567 ts_tablet_manager.cc:1918] T a67fe85812a84af5a28d26b96b49cd15 P 479cdfbbfa0a496abd505220830d412e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 08:44:47.375594 8567 log.cc:1196] T a67fe85812a84af5a28d26b96b49cd15 P 479cdfbbfa0a496abd505220830d412e: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790585022442112-388-0/minicluster-data/ts-1/wal/wals/a67fe85812a84af5a28d26b96b49cd15
I20260928 08:44:47.376071 8567 ts_tablet_manager.cc:1939] T a67fe85812a84af5a28d26b96b49cd15 P 479cdfbbfa0a496abd505220830d412e: Deleting consensus metadata
I20260928 08:44:47.376938 7969 catalog_manager.cc:4841] TS 479cdfbbfa0a496abd505220830d412e (127.0.97.2:40415): tablet a67fe85812a84af5a28d26b96b49cd15 (table test-table [id=ef9e07fadd214a4bbb2f917e6a9bdbbc]) successfully deleted
Sep 28 08:44:47 dist-test-slave-n6xs krb5kdc[7940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.1@KRBTEST.COM
I20260928 08:44:47.388789 8598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:47.361572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46112 (local address 127.0.97.1:40955)
0928 08:44:47.361857 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:47.361866 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:47.372308 (+ 10442us) server_negotiation.cc:407] Connection header received
0928 08:44:47.372599 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:47.372604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:47.372695 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:47.372815 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:47.374484 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:47.375630 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:47.377518 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:47.378055 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:47.378204 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:47.382759 (+ 4555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:47.382771 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:47.382785 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:47.382813 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:47.385955 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:47.386611 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:47.386619 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:47.386621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:47.386749 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:47.387160 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:47.387164 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:47.387166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:47.387499 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:47.387838 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:47.388179 (+ 341us) server_negotiation.cc:299] Negotiation successful
0928 08:44:47.388400 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 08:44:47.389442 8187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:47.501005 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 8056
I20260928 08:44:47.520583 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 8196
Sep 28 08:44:47 dist-test-slave-n6xs krb5kdc[7940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.2@KRBTEST.COM
I20260928 08:44:47.535332 8537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:47.521346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53294 (local address 127.0.97.2:40415)
0928 08:44:47.521524 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:47.521529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:47.521547 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:44:47.521737 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:47.521741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:47.521798 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:47.521877 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:47.523028 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:47.523656 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:47.524965 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:47.525513 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:47.525631 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:47.529615 (+ 3984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:47.529628 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:47.529643 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:47.529683 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:47.532916 (+ 3233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:47.533522 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:47.533530 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:47.533532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:47.533677 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:47.533989 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:47.533992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:47.533995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:47.534356 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:47.534585 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:47.534809 (+ 224us) server_negotiation.cc:299] Negotiation successful
0928 08:44:47.534954 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 08:44:47.536137 8326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:47.658509 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 8196
I20260928 08:44:47.681056 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 8335
Sep 28 08:44:47 dist-test-slave-n6xs krb5kdc[7940](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.97.3@KRBTEST.COM
I20260928 08:44:47.695760 8545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:47.681820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37962 (local address 127.0.97.3:34497)
0928 08:44:47.681998 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:47.682005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:47.682105 (+ 100us) server_negotiation.cc:407] Connection header received
0928 08:44:47.682321 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:47.682325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:47.682394 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:47.682545 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:47.683755 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:47.684495 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:47.685673 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:47.686076 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:47.686208 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:47.690266 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:47.690278 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:47.690295 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:47.690324 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:47.693092 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:47.693718 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:47.693726 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:47.693728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:47.693859 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:47.694237 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:47.694243 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:47.694246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:47.694581 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:47.694905 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:47.695201 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 08:44:47.695443 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 08:44:47.696508 8465 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:47.818871 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 8335
I20260928 08:44:47.840319 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 7953
I20260928 08:44:47.841507 8025 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:47.955307 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 7953
2026-09-28T08:44:47Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5658 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:44:48 dist-test-slave-n6xs krb5kdc[8618](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:44:48 dist-test-slave-n6xs krb5kdc[8618](info): set up 2 sockets
Sep 28 08:44:48 dist-test-slave-n6xs krb5kdc[8618](info): commencing operation
krb5kdc: starting...
W20260928 08:44:49.057080 388 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:44:49 dist-test-slave-n6xs krb5kdc[8618](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:44:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:44:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:44:49.267519 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39943
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:39943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:49.596926 8631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:49.597289 8631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:49.597469 8631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:49.605472 8631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:44:49.605639 8631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:49.605764 8631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:49.605877 8631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:44:49.605964 8631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:44:49.616843 8631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:39943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:39943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:49.618186 8631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:49.619413 8631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:49.631068 8636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:49.631034 8639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:49.631294 8637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:49.632450 8631 server_base.cc:1034] running on GCE node
I20260928 08:44:49.633319 8631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:49.634622 8631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:49.635928 8631 hybrid_clock.cc:648] HybridClock initialized: now 1790585089635891 us; error 51 us; skew 500 ppm
I20260928 08:44:49.636300 8631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T08:44:49Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1748 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:44:49 dist-test-slave-n6xs krb5kdc[8645](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:44:49 dist-test-slave-n6xs krb5kdc[8645](info): set up 2 sockets
Sep 28 08:44:49 dist-test-slave-n6xs krb5kdc[8645](info): commencing operation
krb5kdc: starting...
W20260928 08:44:50.816933 388 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:44:50 dist-test-slave-n6xs krb5kdc[8645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:44:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:44:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:44:51.040721 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42245
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42245
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:51.367621 8658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:51.368027 8658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:51.368209 8658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:51.375838 8658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:44:51.375996 8658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:51.376117 8658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:51.376228 8658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:44:51.376320 8658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:44:51.386579 8658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:51.388057 8658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:51.389424 8658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:51.402000 8663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:51.401960 8664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:51.402045 8666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:51.402745 8658 server_base.cc:1034] running on GCE node
I20260928 08:44:51.403548 8658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:51.405144 8658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:51.406374 8658 hybrid_clock.cc:648] HybridClock initialized: now 1790585091406324 us; error 33 us; skew 500 ppm
I20260928 08:44:51.406769 8658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:51 dist-test-slave-n6xs krb5kdc[8645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585091, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:51.410069 8658 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:44:51.411942 8658 webserver.cc:460] Webserver started at http://127.0.97.62:36549/ using document root <none> and password file <none>
I20260928 08:44:51.413189 8658 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:51.413378 8658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:51.413828 8658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:51.419445 8658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "12dbc24cbe764895bc0c72ac2c47fc0e"
format_stamp: "Formatted at 2026-09-28 08:44:51 on dist-test-slave-n6xs"
server_key: "8a16e8a9907a578074081e3bf588ed43"
server_key_iv: "34ccb6645a49c07b22b70fc4cf1c2ebf"
server_key_version: "encryptionkey@0"
I20260928 08:44:51.420825 8658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "12dbc24cbe764895bc0c72ac2c47fc0e"
format_stamp: "Formatted at 2026-09-28 08:44:51 on dist-test-slave-n6xs"
server_key: "8a16e8a9907a578074081e3bf588ed43"
server_key_iv: "34ccb6645a49c07b22b70fc4cf1c2ebf"
server_key_version: "encryptionkey@0"
I20260928 08:44:51.429800 8658 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20260928 08:44:51.434727 8673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:51.436638 8658 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 08:44:51.436832 8658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "12dbc24cbe764895bc0c72ac2c47fc0e"
format_stamp: "Formatted at 2026-09-28 08:44:51 on dist-test-slave-n6xs"
server_key: "8a16e8a9907a578074081e3bf588ed43"
server_key_iv: "34ccb6645a49c07b22b70fc4cf1c2ebf"
server_key_version: "encryptionkey@0"
I20260928 08:44:51.437100 8658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:51.469182 8658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:51.473459 8658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:51.473716 8658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:51.492537 8658 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42245
I20260928 08:44:51.492568 8724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42245 every 8 connection(s)
I20260928 08:44:51.494000 8658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:44:51.497609 8725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:44:51.503516 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 8658
I20260928 08:44:51.503734 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:44:51.504420 388 external_mini_cluster.cc:1428] Setting key a03cc283ba507daa5e223411dfa2c769
I20260928 08:44:51.508857 8725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e: Bootstrap starting.
I20260928 08:44:51.511854 8725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e: Neither blocks nor log segments found. Creating new log.
I20260928 08:44:51.512830 8725 log.cc:824] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e: Log is configured to *not* fsync() on all Append() calls
I20260928 08:44:51.515517 8725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e: No bootstrap required, opened a new log
Sep 28 08:44:51 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:44:51.524339 8725 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12dbc24cbe764895bc0c72ac2c47fc0e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42245 } }
I20260928 08:44:51.524775 8725 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:44:51.524916 8725 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12dbc24cbe764895bc0c72ac2c47fc0e, State: Initialized, Role: FOLLOWER
I20260928 08:44:51.525553 8725 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12dbc24cbe764895bc0c72ac2c47fc0e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42245 } }
I20260928 08:44:51.525869 8725 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:44:51.526013 8725 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:44:51.526191 8725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:44:51.527953 8728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:51.506510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53548 (local address 127.0.97.62:42245)
0928 08:44:51.507294 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:51.507306 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:51.507344 (+ 38us) server_negotiation.cc:407] Connection header received
0928 08:44:51.509119 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:51.509141 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:51.509515 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:51.509898 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:51.511417 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:51.512834 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:51.514548 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:51.515184 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:51.515408 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:51.519758 (+ 4350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:51.519767 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:51.519782 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:51.519915 (+ 133us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:51.523374 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:51.524193 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:51.524199 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:51.524205 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:51.524376 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:51.524945 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:51.524950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:51.524953 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:51.525543 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:51.525813 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:51.526357 (+ 544us) server_negotiation.cc:299] Negotiation successful
0928 08:44:51.526840 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":369,"threads_started":1}
I20260928 08:44:51.528753 8725 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12dbc24cbe764895bc0c72ac2c47fc0e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42245 } }
I20260928 08:44:51.529306 8725 leader_election.cc:302] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12dbc24cbe764895bc0c72ac2c47fc0e; no voters:
I20260928 08:44:51.530505 8725 leader_election.cc:288] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:44:51.530630 8730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:44:51.531809 8730 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e [term 1 LEADER]: Becoming Leader. State: Replica: 12dbc24cbe764895bc0c72ac2c47fc0e, State: Running, Role: LEADER
I20260928 08:44:51.532346 8730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12dbc24cbe764895bc0c72ac2c47fc0e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42245 } }
I20260928 08:44:51.532872 8725 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:44:51.535342 8732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12dbc24cbe764895bc0c72ac2c47fc0e. Latest consensus state: current_term: 1 leader_uuid: "12dbc24cbe764895bc0c72ac2c47fc0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12dbc24cbe764895bc0c72ac2c47fc0e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42245 } } }
I20260928 08:44:51.535368 8731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12dbc24cbe764895bc0c72ac2c47fc0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12dbc24cbe764895bc0c72ac2c47fc0e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42245 } } }
I20260928 08:44:51.535864 8731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:51.535866 8732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:51.540540 8736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:44:51.545738 8736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:44:51.554219 8736 catalog_manager.cc:1347] Generated new cluster ID: 56423591fa994bb1920336355ffee6da
I20260928 08:44:51.554338 8736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:44:51.573248 8736 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:44:51.574560 8736 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:44:51.587385 8736 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e: Generated new TSK 0
I20260928 08:44:51.588358 8736 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:44:51.647439 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42245
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:51.978616 8751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:51.978940 8751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:51.979122 8751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:51.986613 8751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:51.986739 8751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:51.986936 8751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:44:51.997074 8751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:51.998409 8751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:51.999881 8751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:52.013300 8756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:52.013343 8759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:52.013383 8757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:52.014591 8751 server_base.cc:1034] running on GCE node
I20260928 08:44:52.015228 8751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:52.016296 8751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:52.017542 8751 hybrid_clock.cc:648] HybridClock initialized: now 1790585092017477 us; error 53 us; skew 500 ppm
I20260928 08:44:52.017879 8751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:52 dist-test-slave-n6xs krb5kdc[8645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:52.024507 8751 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:44:52.026080 8751 webserver.cc:460] Webserver started at http://127.0.97.1:35451/ using document root <none> and password file <none>
I20260928 08:44:52.026780 8751 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:52.026923 8751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:52.027256 8751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:52.030262 8751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-0/data/instance:
uuid: "27926459b50e44b8b9eecef826ddba68"
format_stamp: "Formatted at 2026-09-28 08:44:52 on dist-test-slave-n6xs"
server_key: "258d95017290c309a4883629d364e8bb"
server_key_iv: "451fc73ee45a327c8be169add18495b3"
server_key_version: "encryptionkey@0"
I20260928 08:44:52.031037 8751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-0/wal/instance:
uuid: "27926459b50e44b8b9eecef826ddba68"
format_stamp: "Formatted at 2026-09-28 08:44:52 on dist-test-slave-n6xs"
server_key: "258d95017290c309a4883629d364e8bb"
server_key_iv: "451fc73ee45a327c8be169add18495b3"
server_key_version: "encryptionkey@0"
I20260928 08:44:52.036412 8751 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260928 08:44:52.039790 8766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:52.041337 8751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:44:52.041527 8751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-0/wal
uuid: "27926459b50e44b8b9eecef826ddba68"
format_stamp: "Formatted at 2026-09-28 08:44:52 on dist-test-slave-n6xs"
server_key: "258d95017290c309a4883629d364e8bb"
server_key_iv: "451fc73ee45a327c8be169add18495b3"
server_key_version: "encryptionkey@0"
I20260928 08:44:52.041723 8751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:52.074229 8751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:52.078693 8751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:52.079051 8751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:52.080137 8751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:44:52.081506 8751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:44:52.081625 8751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:52.081784 8751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:44:52.081873 8751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:52.117774 8751 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39403
I20260928 08:44:52.117806 8878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39403 every 8 connection(s)
I20260928 08:44:52.119089 8751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:44:52.121202 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 8751
I20260928 08:44:52.121407 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:44:52.122159 388 external_mini_cluster.cc:1428] Setting key 0fa7bf2b58bae9238ea21c03f94ec291
Sep 28 08:44:52 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:44:52.139744 8881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:52.121934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53193 (local address 127.0.97.62:42245)
0928 08:44:52.122471 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:52.122534 (+ 63us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:52.123509 (+ 975us) server_negotiation.cc:407] Connection header received
0928 08:44:52.125509 (+ 2000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:52.125514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:52.125594 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:52.125734 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:52.127189 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:52.127963 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:52.129494 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:52.130185 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:52.130331 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:52.134174 (+ 3843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:52.134179 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:52.134182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:52.134211 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:52.136836 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:52.137534 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:52.137542 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:52.137545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:52.137660 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:52.138115 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:52.138122 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:52.138125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:52.138363 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:52.138532 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:52.139182 (+ 650us) server_negotiation.cc:299] Negotiation successful
0928 08:44:52.139387 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":319,"threads_started":1}
I20260928 08:44:52.140782 8882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:52.122517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.1:53193)
0928 08:44:52.123322 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:52.123370 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:52.125252 (+ 1882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:52.125979 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:52.125987 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:52.126479 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:52.127029 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:52.127045 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:52.128127 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:52.128136 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:52.129334 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:52.129346 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:52.130424 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:52.130430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:52.130640 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:52.131301 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:52.131322 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:52.133936 (+ 2614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:52.136985 (+ 3049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:52.136993 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:52.137008 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:52.137409 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:52.137788 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:52.137795 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:52.137798 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:52.137993 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:52.138488 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:52.138497 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:52.138815 (+ 318us) client_negotiation.cc:769] Sending connection context
0928 08:44:52.139088 (+ 273us) client_negotiation.cc:240] Negotiation successful
0928 08:44:52.139579 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":329,"threads_started":1}
I20260928 08:44:52.142427 8879 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42245
I20260928 08:44:52.142732 8879 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 08:44:52.143519 8879 heartbeater.cc:511] Master 127.0.97.62:42245 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260928 08:44:52.145905 8690 ts_manager.cc:194] Registered new tserver with Master: 27926459b50e44b8b9eecef826ddba68 (127.0.97.1:39403)
I20260928 08:44:52.147454 8690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:53193
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 08:44:52.172113 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42245
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:52.489351 8885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:52.489605 8885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:52.489759 8885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:52.499854 8885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:52.499985 8885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:52.500154 8885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:44:52.515167 8885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:52.516505 8885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:52.517676 8885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:52.530858 8891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:52.530869 8893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:52.531044 8890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:52.531574 8885 server_base.cc:1034] running on GCE node
I20260928 08:44:52.532233 8885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:52.533209 8885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:52.534430 8885 hybrid_clock.cc:648] HybridClock initialized: now 1790585092534370 us; error 56 us; skew 500 ppm
I20260928 08:44:52.534745 8885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:52 dist-test-slave-n6xs krb5kdc[8645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:52.537813 8885 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:44:52.539300 8885 webserver.cc:460] Webserver started at http://127.0.97.2:43145/ using document root <none> and password file <none>
I20260928 08:44:52.539965 8885 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:52.540073 8885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:52.540362 8885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:52.543221 8885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-1/data/instance:
uuid: "3a8b035100b447b7b805d80dc97026c9"
format_stamp: "Formatted at 2026-09-28 08:44:52 on dist-test-slave-n6xs"
server_key: "991dfa26b4981e53be09e49ea0599dd1"
server_key_iv: "e41d4d7bc522538e7728db5aaf76def4"
server_key_version: "encryptionkey@0"
I20260928 08:44:52.544081 8885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-1/wal/instance:
uuid: "3a8b035100b447b7b805d80dc97026c9"
format_stamp: "Formatted at 2026-09-28 08:44:52 on dist-test-slave-n6xs"
server_key: "991dfa26b4981e53be09e49ea0599dd1"
server_key_iv: "e41d4d7bc522538e7728db5aaf76def4"
server_key_version: "encryptionkey@0"
I20260928 08:44:52.549053 8885 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 08:44:52.552378 8900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:52.553766 8885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:44:52.553979 8885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-1/wal
uuid: "3a8b035100b447b7b805d80dc97026c9"
format_stamp: "Formatted at 2026-09-28 08:44:52 on dist-test-slave-n6xs"
server_key: "991dfa26b4981e53be09e49ea0599dd1"
server_key_iv: "e41d4d7bc522538e7728db5aaf76def4"
server_key_version: "encryptionkey@0"
I20260928 08:44:52.554237 8885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:52.574534 8885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:52.578890 8885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:52.579190 8885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:52.580168 8885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:44:52.581449 8885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:44:52.581529 8885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:52.581670 8885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:44:52.581733 8885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:52.612632 8885 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39889
I20260928 08:44:52.612663 9012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39889 every 8 connection(s)
I20260928 08:44:52.613881 8885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:44:52.622341 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 8885
I20260928 08:44:52.622562 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:44:52.622965 388 external_mini_cluster.cc:1428] Setting key b337d00c9eb234799423ceb48a73b7fb
Sep 28 08:44:52 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:44:52.635761 8881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:52.616717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54965 (local address 127.0.97.62:42245)
0928 08:44:52.616927 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:52.616932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:52.618313 (+ 1381us) server_negotiation.cc:407] Connection header received
0928 08:44:52.620227 (+ 1914us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:52.620232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:52.620325 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:52.620439 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:52.621893 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:52.622662 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:52.624369 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:52.624976 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:52.625108 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:52.629099 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:52.629105 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:52.629108 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:52.629141 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:52.632349 (+ 3208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:52.633173 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:52.633179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:52.633182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:52.633365 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:52.633839 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:52.633845 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:52.633848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:52.634225 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:52.634402 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:52.635123 (+ 721us) server_negotiation.cc:299] Negotiation successful
0928 08:44:52.635372 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 08:44:52.636775 9015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:52.617179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.2:54965)
0928 08:44:52.618055 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:52.618093 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:52.619969 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:52.620688 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:52.620696 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:52.621176 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:52.621724 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:52.621741 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:52.622832 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:52.622836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:52.624214 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:52.624225 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:52.625194 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:52.625201 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:52.625458 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:52.626261 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:52.626288 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:52.628838 (+ 2550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:52.632505 (+ 3667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:52.632515 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:52.632532 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:52.633033 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:52.633490 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:52.633496 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:52.633500 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:52.633707 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:52.634385 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:52.634394 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:52.634739 (+ 345us) client_negotiation.cc:769] Sending connection context
0928 08:44:52.635020 (+ 281us) client_negotiation.cc:240] Negotiation successful
0928 08:44:52.635567 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":287,"threads_started":1}
I20260928 08:44:52.638327 9013 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42245
I20260928 08:44:52.638643 9013 heartbeater.cc:464] Registering TS with master...
I20260928 08:44:52.639322 9013 heartbeater.cc:511] Master 127.0.97.62:42245 requested a full tablet report, sending...
I20260928 08:44:52.640821 8690 ts_manager.cc:194] Registered new tserver with Master: 3a8b035100b447b7b805d80dc97026c9 (127.0.97.2:39889)
I20260928 08:44:52.641718 8690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:54965
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:44:52.674974 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42245
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:52.991205 9018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:52.991472 9018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:52.991566 9018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:52.998579 9018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:52.998685 9018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:52.998870 9018 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:44:53.008239 9018 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:53.009521 9018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:53.010776 9018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:53.023912 9023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:53.023985 9024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:53.023952 9026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:53.024962 9018 server_base.cc:1034] running on GCE node
I20260928 08:44:53.025514 9018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:53.026412 9018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:53.027630 9018 hybrid_clock.cc:648] HybridClock initialized: now 1790585093027501 us; error 125 us; skew 500 ppm
I20260928 08:44:53.027983 9018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:53 dist-test-slave-n6xs krb5kdc[8645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:53.031430 9018 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:44:53.032860 9018 webserver.cc:460] Webserver started at http://127.0.97.3:45275/ using document root <none> and password file <none>
I20260928 08:44:53.033603 9018 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:53.033754 9018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:53.034018 9018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:44:53.036929 9018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-2/data/instance:
uuid: "b3a34e9fb044412ca6501ae425238c93"
format_stamp: "Formatted at 2026-09-28 08:44:53 on dist-test-slave-n6xs"
server_key: "3a7fd2f8fe032c77886cc9c7230ba546"
server_key_iv: "6969aa8684921cfd19ef0536d5a40882"
server_key_version: "encryptionkey@0"
I20260928 08:44:53.037619 9018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-2/wal/instance:
uuid: "b3a34e9fb044412ca6501ae425238c93"
format_stamp: "Formatted at 2026-09-28 08:44:53 on dist-test-slave-n6xs"
server_key: "3a7fd2f8fe032c77886cc9c7230ba546"
server_key_iv: "6969aa8684921cfd19ef0536d5a40882"
server_key_version: "encryptionkey@0"
I20260928 08:44:53.042858 9018 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:44:53.046510 9033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:53.048058 9018 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:44:53.048192 9018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-2/wal
uuid: "b3a34e9fb044412ca6501ae425238c93"
format_stamp: "Formatted at 2026-09-28 08:44:53 on dist-test-slave-n6xs"
server_key: "3a7fd2f8fe032c77886cc9c7230ba546"
server_key_iv: "6969aa8684921cfd19ef0536d5a40882"
server_key_version: "encryptionkey@0"
I20260928 08:44:53.048344 9018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:53.078855 9018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:53.082617 9018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:53.082854 9018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:53.083714 9018 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:44:53.085057 9018 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:44:53.085124 9018 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:53.085209 9018 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:44:53.085258 9018 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:53.118209 9018 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34697
I20260928 08:44:53.118227 9145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34697 every 8 connection(s)
I20260928 08:44:53.119462 9018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:44:53.125365 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 9018
I20260928 08:44:53.125571 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:44:53.126075 388 external_mini_cluster.cc:1428] Setting key 1055f8d2d429065da246e3ed09218f6c
Sep 28 08:44:53 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:44:53.138797 8881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:53.122068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57125 (local address 127.0.97.62:42245)
0928 08:44:53.122260 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:53.122265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:53.123334 (+ 1069us) server_negotiation.cc:407] Connection header received
0928 08:44:53.124977 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:53.124982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:53.125064 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:53.125151 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:53.127085 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:53.128143 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:53.129800 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:53.130168 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:53.130266 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:53.133533 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:53.133538 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:53.133541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:53.133571 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:53.136187 (+ 2616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:53.136841 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:53.136845 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:53.136847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:53.136960 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:53.137424 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:53.137427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:53.137429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:53.137663 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:53.137783 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:53.138403 (+ 620us) server_negotiation.cc:299] Negotiation successful
0928 08:44:53.138538 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 08:44:53.140027 9148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:53.122476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.3:57125)
0928 08:44:53.123174 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:53.123208 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:53.124747 (+ 1539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:53.125449 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:53.125462 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:53.126160 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:53.126913 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:53.126936 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:53.128276 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:53.128281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:53.129684 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:53.129695 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:53.130351 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:53.130356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:53.130554 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:53.131156 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:53.131177 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:53.133365 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:44:53.136309 (+ 2944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:53.136315 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:53.136326 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:53.136738 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:53.137074 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:44:53.137077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:44:53.137079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:44:53.137306 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:44:53.137801 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:44:53.137806 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:44:53.138094 (+ 288us) client_negotiation.cc:769] Sending connection context
0928 08:44:53.138317 (+ 223us) client_negotiation.cc:240] Negotiation successful
0928 08:44:53.138694 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":282,"threads_started":1}
I20260928 08:44:53.141649 9146 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42245
I20260928 08:44:53.142028 9146 heartbeater.cc:464] Registering TS with master...
I20260928 08:44:53.142817 9146 heartbeater.cc:511] Master 127.0.97.62:42245 requested a full tablet report, sending...
I20260928 08:44:53.144484 8689 ts_manager.cc:194] Registered new tserver with Master: b3a34e9fb044412ca6501ae425238c93 (127.0.97.3:34697)
I20260928 08:44:53.145531 8689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57125
I20260928 08:44:53.150363 8879 heartbeater.cc:503] Master 127.0.97.62:42245 was elected leader, sending a full tablet report...
I20260928 08:44:53.153777 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/oryx_127.0.97.1.keytab.
Entry for principal oryx/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/oryx_127.0.97.1.keytab.
WARNING: no policy specified for oryx/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/oryx_127.0.97.2.keytab.
Entry for principal oryx/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/oryx_127.0.97.2.keytab.
WARNING: no policy specified for oryx/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/oryx_127.0.97.3.keytab.
Entry for principal oryx/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/oryx_127.0.97.3.keytab.
I20260928 08:44:53.291682 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 8751
Sep 28 08:44:53 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:44:53.309074 9156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:53.292988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34182 (local address 127.0.97.1:39403)
0928 08:44:53.293727 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:53.293734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:53.293760 (+ 26us) server_negotiation.cc:407] Connection header received
0928 08:44:53.293819 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:53.293825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:53.294059 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:53.294196 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:53.295421 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:53.296564 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:53.297926 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:53.298501 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:53.298624 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:53.302665 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:53.302680 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:53.302695 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:53.302734 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:53.305971 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:53.306765 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:53.306773 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:53.306776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:53.306916 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:53.307355 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:53.307360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:53.307362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:53.307723 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:53.308044 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:53.308377 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 08:44:53.308715 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":384,"threads_started":1}
I20260928 08:44:53.310411 8874 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:53.395771 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 8751
I20260928 08:44:53.412540 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 8885
Sep 28 08:44:53 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 08:44:53.431067 9158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:53.413674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34600 (local address 127.0.97.2:39889)
0928 08:44:53.414388 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:53.414400 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:53.414433 (+ 33us) server_negotiation.cc:407] Connection header received
0928 08:44:53.414512 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:53.414521 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:53.414825 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:53.415021 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:53.416766 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:53.418382 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:53.420066 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:53.421048 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:53.421174 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:53.425776 (+ 4602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:53.425786 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:53.425800 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:53.425838 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:53.428676 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:53.429286 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:53.429293 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:53.429295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:53.429412 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:53.429763 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:53.429767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:53.429769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:53.430102 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:53.430332 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:53.430560 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 08:44:53.430768 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":365,"threads_started":1}
I20260928 08:44:53.432428 9008 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:53.515522 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 8885
I20260928 08:44:53.531703 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 9018
Sep 28 08:44:53 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 08:44:53.549202 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:53.532831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56380 (local address 127.0.97.3:34697)
0928 08:44:53.533380 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:53.533387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:53.533414 (+ 27us) server_negotiation.cc:407] Connection header received
0928 08:44:53.533503 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:53.533510 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:53.533784 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:53.533934 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:53.535234 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:53.536486 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:53.537796 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:53.538358 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:53.538468 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:53.543156 (+ 4688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:53.543167 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:53.543180 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:53.543211 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:53.546534 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:53.547229 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:53.547237 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:53.547239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:53.547345 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:53.547681 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:53.547684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:53.547686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:53.548084 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:53.548321 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:53.548632 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 08:44:53.548817 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":252,"threads_started":1}
I20260928 08:44:53.550575 9141 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:53.636660 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 9018
I20260928 08:44:53.652503 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 8658
I20260928 08:44:53.653589 8720 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:44:53.755055 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 8658
I20260928 08:44:53.775367 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42245
--webserver_interface=127.0.97.62
--webserver_port=36549
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42245
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:44:54.095674 9163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:54.096031 9163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:54.096194 9163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:54.103137 9163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:44:54.103272 9163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:54.103382 9163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:54.103495 9163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:44:54.103580 9163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:44:54.112987 9163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=36549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:54.114207 9163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:54.115581 9163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:54.126920 9168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:54.126894 9169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:54.127205 9171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:54.128263 9163 server_base.cc:1034] running on GCE node
I20260928 08:44:54.129173 9163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:54.130688 9163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:54.131911 9163 hybrid_clock.cc:648] HybridClock initialized: now 1790585094131861 us; error 41 us; skew 500 ppm
I20260928 08:44:54.132196 9163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:54 dist-test-slave-n6xs krb5kdc[8645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:54.135249 9163 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:44:54.136802 9163 webserver.cc:460] Webserver started at http://127.0.97.62:36549/ using document root <none> and password file <none>
I20260928 08:44:54.137455 9163 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:54.137578 9163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:54.143261 9163 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 08:44:54.146147 9178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:54.147475 9163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 08:44:54.147665 9163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "12dbc24cbe764895bc0c72ac2c47fc0e"
format_stamp: "Formatted at 2026-09-28 08:44:51 on dist-test-slave-n6xs"
server_key: "8a16e8a9907a578074081e3bf588ed43"
server_key_iv: "34ccb6645a49c07b22b70fc4cf1c2ebf"
server_key_version: "encryptionkey@0"
I20260928 08:44:54.148717 9163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:54.160342 9163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:54.164028 9163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:54.164273 9163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:54.182961 9163 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42245
I20260928 08:44:54.183054 9229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42245 every 8 connection(s)
I20260928 08:44:54.184435 9163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:44:54.186075 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 9163
I20260928 08:44:54.186846 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:39403
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:42245
--webserver_port=35451
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.1
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/oryx_127.0.97.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:44:54.191215 9230 sys_catalog.cc:263] Verifying existing consensus state
I20260928 08:44:54.195016 9230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e: Bootstrap starting.
I20260928 08:44:54.213639 9230 log.cc:824] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e: Log is configured to *not* fsync() on all Append() calls
I20260928 08:44:54.221946 9230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:44:54.222456 9230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e: Bootstrap complete.
I20260928 08:44:54.231150 9230 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12dbc24cbe764895bc0c72ac2c47fc0e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42245 } }
I20260928 08:44:54.232020 9230 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12dbc24cbe764895bc0c72ac2c47fc0e, State: Initialized, Role: FOLLOWER
I20260928 08:44:54.232589 9230 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12dbc24cbe764895bc0c72ac2c47fc0e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42245 } }
I20260928 08:44:54.232820 9230 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:44:54.232925 9230 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:44:54.233119 9230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e [term 1 FOLLOWER]: Advancing to term 2
I20260928 08:44:54.236694 9230 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12dbc24cbe764895bc0c72ac2c47fc0e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42245 } }
I20260928 08:44:54.237185 9230 leader_election.cc:302] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12dbc24cbe764895bc0c72ac2c47fc0e; no voters:
I20260928 08:44:54.237819 9230 leader_election.cc:288] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 08:44:54.237951 9234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e [term 2 FOLLOWER]: Leader election won for term 2
I20260928 08:44:54.239066 9234 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e [term 2 LEADER]: Becoming Leader. State: Replica: 12dbc24cbe764895bc0c72ac2c47fc0e, State: Running, Role: LEADER
I20260928 08:44:54.239620 9234 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12dbc24cbe764895bc0c72ac2c47fc0e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42245 } }
I20260928 08:44:54.239943 9230 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:44:54.242586 9236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12dbc24cbe764895bc0c72ac2c47fc0e. Latest consensus state: current_term: 2 leader_uuid: "12dbc24cbe764895bc0c72ac2c47fc0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12dbc24cbe764895bc0c72ac2c47fc0e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42245 } } }
I20260928 08:44:54.242951 9236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:54.243516 9235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "12dbc24cbe764895bc0c72ac2c47fc0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12dbc24cbe764895bc0c72ac2c47fc0e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42245 } } }
I20260928 08:44:54.243841 9235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e [sys.catalog]: This master's current role is: LEADER
I20260928 08:44:54.245900 9240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:44:54.251358 9240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:44:54.252835 9240 catalog_manager.cc:1259] Loaded cluster ID: 56423591fa994bb1920336355ffee6da
I20260928 08:44:54.252980 9240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:44:54.257066 9240 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:44:54.258692 9240 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 12dbc24cbe764895bc0c72ac2c47fc0e: Loaded TSK: 0
I20260928 08:44:54.260170 9240 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 08:44:54.522783 9232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:54.523073 9232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:54.523233 9232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:54.530375 9232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:54.530474 9232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:54.530651 9232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:44:54.540277 9232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/oryx_127.0.97.1.keytab
--principal=oryx/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:39403
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=35451
--tserver_master_addrs=127.0.97.62:42245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:54.541570 9232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:54.542822 9232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:54.557005 9257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:54.556986 9258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:54.557682 9260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:54.558255 9232 server_base.cc:1034] running on GCE node
I20260928 08:44:54.558892 9232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:54.559755 9232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:54.561028 9232 hybrid_clock.cc:648] HybridClock initialized: now 1790585094560951 us; error 56 us; skew 500 ppm
I20260928 08:44:54.561332 9232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:54 dist-test-slave-n6xs krb5kdc[8645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585094, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:54.564440 9232 init.cc:377] Logged in from keytab as oryx/127.0.97.1@KRBTEST.COM (short username oryx)
I20260928 08:44:54.565881 9232 webserver.cc:460] Webserver started at http://127.0.97.1:35451/ using document root <none> and password file <none>
I20260928 08:44:54.566609 9232 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:54.566745 9232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:54.572067 9232 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 08:44:54.574736 9267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:54.576112 9232 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:44:54.576280 9232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-0/wal
uuid: "27926459b50e44b8b9eecef826ddba68"
format_stamp: "Formatted at 2026-09-28 08:44:52 on dist-test-slave-n6xs"
server_key: "258d95017290c309a4883629d364e8bb"
server_key_iv: "451fc73ee45a327c8be169add18495b3"
server_key_version: "encryptionkey@0"
I20260928 08:44:54.577247 9232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:54.588651 9232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:54.592783 9232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:54.593062 9232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:54.594002 9232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:44:54.595335 9232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:44:54.595463 9232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:54.595595 9232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:44:54.595681 9232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:54.631109 9232 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39403
I20260928 08:44:54.631170 9379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39403 every 8 connection(s)
I20260928 08:44:54.632648 9232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:44:54.637915 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 9232
I20260928 08:44:54.638536 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:39889
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:42245
--webserver_port=43145
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.2
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/oryx_127.0.97.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:44:54 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:44:54 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:44:54.651243 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:54.635834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.1:49159)
0928 08:44:54.636731 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:54.636783 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:54.638506 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:54.639966 (+ 1460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:54.639976 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:54.640632 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:54.641222 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:54.641239 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:54.642837 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:54.642842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:54.644339 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:54.644354 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:54.645648 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:54.645659 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:54.645878 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:54.646556 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:54.646577 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:54.650268 (+ 3691us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":350,"threads_started":1}
W20260928 08:44:54.651859 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:44:54.635845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49159 (local address 127.0.97.62:42245)
0928 08:44:54.636710 (+ 865us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:54.636726 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:54.636908 (+ 182us) server_negotiation.cc:407] Connection header received
0928 08:44:54.639101 (+ 2193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:54.639113 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:54.639416 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:54.639760 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:54.641436 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:54.642682 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:54.644512 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:54.645415 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:54.645617 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:54.650675 (+ 5058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:49159: BlockingRecv error: recv got EOF from 127.0.97.1:49159 (error 108)
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":392,"threads_started":1}
W20260928 08:44:54.652092 9380 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42245 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:42245: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 28 08:44:54 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:44:54 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:44:54.661509 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:54.652665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.1:42573)
0928 08:44:54.652903 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:54.652926 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:54.653002 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:54.653439 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:54.653446 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:54.653827 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:54.654065 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:54.654079 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:54.655590 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:54.655594 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:54.656611 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:54.656623 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:54.657695 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:54.657699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:54.657852 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:54.658460 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:54.658477 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:54.661297 (+ 2820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124}
W20260928 08:44:54.661839 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:44:54.652758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42573 (local address 127.0.97.62:42245)
0928 08:44:54.652946 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:54.652954 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:54.652975 (+ 21us) server_negotiation.cc:407] Connection header received
0928 08:44:54.653107 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:54.653115 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:54.653196 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:54.653303 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:54.654240 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:54.655430 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:54.656757 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:54.657527 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:54.657668 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:54.661565 (+ 3897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42573: BlockingRecv error: recv got EOF from 127.0.97.1:42573 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 28 08:44:54 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:44:54 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:44:54.669916 9382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:54.662256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.1:53137)
0928 08:44:54.662583 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:54.662600 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:54.662693 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:54.662993 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:54.662996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:54.663261 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:54.663390 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:54.663397 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:54.664771 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:54.664778 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:54.665646 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:54.665656 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:54.666504 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:54.666507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:54.666636 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:54.667178 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:54.667193 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:54.669734 (+ 2541us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204}
W20260928 08:44:54.670193 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:44:54.662493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53137 (local address 127.0.97.62:42245)
0928 08:44:54.662670 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:54.662675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:54.662694 (+ 19us) server_negotiation.cc:407] Connection header received
0928 08:44:54.662772 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:54.662778 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:54.662865 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:54.662970 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:54.663511 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:54.664599 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:54.665767 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:54.666383 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:54.666497 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:54.669957 (+ 3460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53137: BlockingRecv error: recv got EOF from 127.0.97.1:53137 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 08:44:54.670341 9380 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 08:44:54.962322 9384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:54.962630 9384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:54.962850 9384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:54.970432 9384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:54.970577 9384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:54.970799 9384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:44:54.980904 9384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/oryx_127.0.97.2.keytab
--principal=oryx/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:39889
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=43145
--tserver_master_addrs=127.0.97.62:42245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:54.982618 9384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:54.984099 9384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:54.998989 9389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:54.998958 9392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:54.999020 9390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:54.999765 9384 server_base.cc:1034] running on GCE node
I20260928 08:44:55.000355 9384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:55.001308 9384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:55.002556 9384 hybrid_clock.cc:648] HybridClock initialized: now 1790585095002521 us; error 59 us; skew 500 ppm
I20260928 08:44:55.002861 9384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:55 dist-test-slave-n6xs krb5kdc[8645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585095, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:55.005928 9384 init.cc:377] Logged in from keytab as oryx/127.0.97.2@KRBTEST.COM (short username oryx)
I20260928 08:44:55.007393 9384 webserver.cc:460] Webserver started at http://127.0.97.2:43145/ using document root <none> and password file <none>
I20260928 08:44:55.008172 9384 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:55.008325 9384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:55.013496 9384 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 08:44:55.016279 9399 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:55.017663 9384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:44:55.017849 9384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-1/wal
uuid: "3a8b035100b447b7b805d80dc97026c9"
format_stamp: "Formatted at 2026-09-28 08:44:52 on dist-test-slave-n6xs"
server_key: "991dfa26b4981e53be09e49ea0599dd1"
server_key_iv: "e41d4d7bc522538e7728db5aaf76def4"
server_key_version: "encryptionkey@0"
I20260928 08:44:55.018823 9384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:55.036279 9384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:55.040452 9384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:55.040746 9384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:55.041724 9384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:44:55.042991 9384 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:44:55.043119 9384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:55.043252 9384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:44:55.043345 9384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:55.077692 9384 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:39889
I20260928 08:44:55.077802 9511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:39889 every 8 connection(s)
I20260928 08:44:55.079114 9384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:44:55.082333 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 9384
I20260928 08:44:55.083086 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:34697
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:42245
--webserver_port=45275
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.97.3
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/oryx_127.0.97.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:44:55 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:44:55 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 08:44:55.095765 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:44:55.081840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38885 (local address 127.0.97.62:42245)
0928 08:44:55.082056 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:55.082060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:55.083385 (+ 1325us) server_negotiation.cc:407] Connection header received
0928 08:44:55.085320 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:55.085326 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:55.085416 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:55.085530 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:55.087030 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:55.087753 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:55.090187 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:55.090609 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:55.090749 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:55.095520 (+ 4771us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38885: BlockingRecv error: recv got EOF from 127.0.97.2:38885 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
I20260928 08:44:55.096524 9514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:55.082440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.2:38885)
0928 08:44:55.083220 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:55.083258 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:55.085036 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:55.085759 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:55.085767 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:55.086274 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:55.086847 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:55.086863 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:55.087940 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:55.087945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:55.090039 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:55.090052 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:55.090835 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:55.090843 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:55.091094 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:55.091849 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:55.091884 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:55.095340 (+ 3456us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":303,"threads_started":1}
W20260928 08:44:55.097168 9512 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42245 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:42245: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Sep 28 08:44:55 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:44:55 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:44:55.106489 9514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:55.097846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.2:42877)
0928 08:44:55.098074 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:55.098104 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:55.098184 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:55.098562 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:55.098566 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:55.098882 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:55.099044 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:55.099052 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:55.100613 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:55.100617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:55.101505 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:55.101515 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:55.102580 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:55.102584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:55.102695 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:55.103295 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:55.103310 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:55.106213 (+ 2903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108}
W20260928 08:44:55.106599 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:44:55.097992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42877 (local address 127.0.97.62:42245)
0928 08:44:55.098221 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:55.098229 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:55.098251 (+ 22us) server_negotiation.cc:407] Connection header received
0928 08:44:55.098317 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:55.098325 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:55.098412 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:55.098533 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:55.099193 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:55.100467 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:55.101651 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:55.102434 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:55.102580 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:55.106389 (+ 3809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:42877: BlockingRecv error: recv got EOF from 127.0.97.2:42877 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 28 08:44:55 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:44:55 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:44:55.114660 9514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:55.107220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.2:55245)
0928 08:44:55.107380 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:55.107458 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:55.107555 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:55.107863 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:55.107868 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:55.108166 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:55.108302 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:55.108309 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:55.109257 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:55.109262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:55.110150 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:55.110160 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:55.110829 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:55.110834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:55.111010 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:55.111566 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:55.111582 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:55.114314 (+ 2732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 08:44:55.114706 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:44:55.107374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55245 (local address 127.0.97.62:42245)
0928 08:44:55.107558 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:55.107564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:55.107582 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:44:55.107660 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:55.107664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:55.107723 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:55.107844 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:55.108441 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:55.109123 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:55.110297 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:55.110686 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:55.110819 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:55.114496 (+ 3677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:55245: BlockingRecv error: recv got EOF from 127.0.97.2:55245 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260928 08:44:55.114966 9512 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 08:44:55.409834 9515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:44:55.410179 9515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:44:55.410347 9515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:44:55.420943 9515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:44:55.421097 9515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:44:55.421283 9515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:44:55.431175 9515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/krb5kdc/oryx_127.0.97.3.keytab
--principal=oryx/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:34697
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=45275
--tserver_master_addrs=127.0.97.62:42245
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:44:55.432565 9515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:44:55.433844 9515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:44:55.447783 9520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:55.447853 9523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:44:55.447883 9521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:44:55.448939 9515 server_base.cc:1034] running on GCE node
I20260928 08:44:55.449553 9515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:44:55.450438 9515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:44:55.451640 9515 hybrid_clock.cc:648] HybridClock initialized: now 1790585095451573 us; error 58 us; skew 500 ppm
I20260928 08:44:55.452003 9515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:44:55 dist-test-slave-n6xs krb5kdc[8645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585095, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:44:55.455251 9515 init.cc:377] Logged in from keytab as oryx/127.0.97.3@KRBTEST.COM (short username oryx)
I20260928 08:44:55.456740 9515 webserver.cc:460] Webserver started at http://127.0.97.3:45275/ using document root <none> and password file <none>
I20260928 08:44:55.457415 9515 fs_manager.cc:359] Metadata directory not provided
I20260928 08:44:55.457561 9515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:44:55.462884 9515 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260928 08:44:55.465621 9530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:55.466979 9515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:44:55.467152 9515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-2/wal
uuid: "b3a34e9fb044412ca6501ae425238c93"
format_stamp: "Formatted at 2026-09-28 08:44:53 on dist-test-slave-n6xs"
server_key: "3a7fd2f8fe032c77886cc9c7230ba546"
server_key_iv: "6969aa8684921cfd19ef0536d5a40882"
server_key_version: "encryptionkey@0"
I20260928 08:44:55.468142 9515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:44:55.497596 9515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:44:55.501581 9515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:44:55.501874 9515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:44:55.502835 9515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:44:55.504310 9515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:44:55.504429 9515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:55.504557 9515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:44:55.504652 9515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:44:55.538787 9515 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34697
I20260928 08:44:55.538875 9642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34697 every 8 connection(s)
I20260928 08:44:55.540362 9515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:44:55.545534 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 9515
Sep 28 08:44:55 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:44:55 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 08:44:55.558111 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:44:55.542944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42705 (local address 127.0.97.62:42245)
0928 08:44:55.543116 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:55.543121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:55.544280 (+ 1159us) server_negotiation.cc:407] Connection header received
0928 08:44:55.546551 (+ 2271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:55.546557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:55.546650 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:55.546779 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:55.548699 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:55.549414 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:55.551310 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:55.551903 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:55.552000 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:55.557792 (+ 5792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42705: BlockingRecv error: recv got EOF from 127.0.97.3:42705 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 08:44:55.558656 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:55.543359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.3:42705)
0928 08:44:55.544120 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:55.544159 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:55.546271 (+ 2112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:55.547219 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:55.547232 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:55.547775 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:55.548496 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:55.548521 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:55.549576 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:55.549583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:55.551130 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:55.551145 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:55.552139 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:55.552148 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:55.552432 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:55.553141 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:55.553162 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:55.557568 (+ 4406us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":275,"threads_started":1}
W20260928 08:44:55.559487 9643 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:42245 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.97.62:42245: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
I20260928 08:44:55.562630 9647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:55.546602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39330 (local address 127.0.97.62:42245)
0928 08:44:55.548217 (+ 1615us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:55.548221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:55.548235 (+ 14us) server_negotiation.cc:407] Connection header received
0928 08:44:55.548273 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:55.548276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:55.548322 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:55.548405 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:55.550008 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:55.550741 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:55.552224 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:55.552653 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:55.552788 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:55.555629 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:44:55.555641 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:44:55.555665 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:44:55.555720 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:44:55.559249 (+ 3529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:55.560018 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:55.560025 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:55.560031 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:55.560176 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:44:55.560625 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:44:55.560628 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:44:55.560630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:44:55.561107 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:44:55.561437 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0928 08:44:55.561904 (+ 467us) server_negotiation.cc:299] Negotiation successful
0928 08:44:55.562260 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1529,"thread_start_us":1399,"threads_started":1}
Sep 28 08:44:55 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:44:55 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:44:55.568352 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:55.560087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.3:45007)
0928 08:44:55.560237 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:55.560258 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:55.560349 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:55.560855 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:55.560859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:55.561238 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:55.561449 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:55.561459 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:55.562687 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:55.562693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:55.563773 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:55.563786 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:55.564496 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:55.564501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:55.564677 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:55.565241 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:55.565257 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:55.568166 (+ 2909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 08:44:55.568607 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:44:55.560415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45007 (local address 127.0.97.62:42245)
0928 08:44:55.560598 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:55.560603 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:55.560620 (+ 17us) server_negotiation.cc:407] Connection header received
0928 08:44:55.560667 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:55.560670 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:55.560717 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:55.560837 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:55.561578 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:55.562558 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:55.563936 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:55.564343 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:55.564443 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:55.568341 (+ 3898us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:45007: BlockingRecv error: recv got EOF from 127.0.97.3:45007 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 08:44:55 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:44:55 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:44:55.576232 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:55.568988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.3:60045)
0928 08:44:55.569141 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:55.569181 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:55.569275 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:55.569569 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:55.569573 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:55.569862 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:55.570001 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:55.570008 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:55.570965 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:55.570970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:55.572009 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:55.572019 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:55.572744 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:55.572750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:55.572926 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:55.573463 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:55.573479 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:55.576039 (+ 2560us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 08:44:55.576401 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:44:55.569090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:60045 (local address 127.0.97.62:42245)
0928 08:44:55.569293 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:55.569298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:55.569327 (+ 29us) server_negotiation.cc:407] Connection header received
0928 08:44:55.569373 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:55.569383 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:55.569451 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:55.569548 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:55.570129 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:55.570829 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:55.572136 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:55.572609 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:55.572735 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:55.576189 (+ 3454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:60045: BlockingRecv error: recv got EOF from 127.0.97.3:60045 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
W20260928 08:44:55.576551 9643 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 08:44:55 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:44:55 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:44:55.680318 9648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:55.671107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.1:40551)
0928 08:44:55.671707 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:55.671724 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:55.671860 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:55.672199 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:55.672202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:55.672694 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:55.672875 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:55.672884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:55.674135 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:55.674140 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:55.675545 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:55.675557 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:55.676290 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:55.676294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:55.676452 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:55.677001 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:55.677019 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:55.680008 (+ 2989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":359,"threads_started":1}
W20260928 08:44:55.680509 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:44:55.671132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40551 (local address 127.0.97.62:42245)
0928 08:44:55.671302 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:55.671307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:55.671841 (+ 534us) server_negotiation.cc:407] Connection header received
0928 08:44:55.671964 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:55.671968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:55.672060 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:55.672159 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:55.673041 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:55.673949 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:55.675687 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:55.676142 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:55.676240 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:55.680280 (+ 4040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:40551: BlockingRecv error: recv got EOF from 127.0.97.1:40551 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 28 08:44:56 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:44:56 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:44:56.124099 9649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:56.115614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.2:39691)
0928 08:44:56.116170 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:56.116186 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:56.116295 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:56.116591 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:56.116594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:56.117033 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:56.117224 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:56.117232 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:56.118233 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:56.118237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:56.119202 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:56.119213 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:56.119982 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:56.119986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:56.120100 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:56.120614 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:56.120645 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:56.123793 (+ 3148us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":320,"threads_started":1}
W20260928 08:44:56.124266 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:44:56.115727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39691 (local address 127.0.97.62:42245)
0928 08:44:56.115935 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:56.115939 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:56.116265 (+ 326us) server_negotiation.cc:407] Connection header received
0928 08:44:56.116393 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:56.116396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:56.116464 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:56.116549 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:56.117403 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:56.118098 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:56.119341 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:56.119768 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:56.119900 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:56.124051 (+ 4151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:39691: BlockingRecv error: recv got EOF from 127.0.97.2:39691 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 28 08:44:56 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:44:56 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:44:56.586086 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:56.577263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.3:42533)
0928 08:44:56.577806 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:56.577821 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:56.577996 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:56.578332 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:56.578336 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:56.578945 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:56.579125 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:56.579134 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:56.580395 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:56.580403 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:56.581366 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:56.581376 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:56.582175 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:56.582180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:56.582292 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:56.582918 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:56.582935 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:56.585762 (+ 2827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":305,"threads_started":1}
W20260928 08:44:56.586186 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:44:56.577338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42533 (local address 127.0.97.62:42245)
0928 08:44:56.577577 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:56.577585 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:56.577925 (+ 340us) server_negotiation.cc:407] Connection header received
0928 08:44:56.578117 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:56.578127 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:56.578185 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:56.578303 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:56.579271 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:56.580251 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:56.581543 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:56.582026 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:56.582125 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:56.585952 (+ 3827us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:42533: BlockingRecv error: recv got EOF from 127.0.97.3:42533 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 28 08:44:56 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:44:56 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:44:56.689795 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:56.681311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.1:57267)
0928 08:44:56.681833 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:56.681849 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:56.681963 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:56.682251 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:56.682253 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:56.682669 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:56.682808 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:56.682816 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:56.683887 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:56.683893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:56.684791 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:56.684801 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:56.685514 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:56.685519 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:56.685626 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:56.686522 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:56.686553 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:56.689557 (+ 3004us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":289,"threads_started":1}
W20260928 08:44:56.689950 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:44:56.681349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:57267 (local address 127.0.97.62:42245)
0928 08:44:56.681587 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:56.681591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:56.681941 (+ 350us) server_negotiation.cc:407] Connection header received
0928 08:44:56.682063 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:56.682067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:56.682125 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:56.682215 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:56.682953 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:56.683696 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:56.684926 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:56.685323 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:56.685417 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:56.689728 (+ 4311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:57267: BlockingRecv error: recv got EOF from 127.0.97.1:57267 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 28 08:44:57 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:44:57 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:44:57.137277 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:57.125012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.2:54073)
0928 08:44:57.125614 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:57.125631 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:57.125749 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:57.126109 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:57.126113 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:57.126495 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:57.126646 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:57.126654 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:57.127920 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:57.127925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:57.128825 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:57.128836 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:57.129748 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:57.129754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:57.129923 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:57.134101 (+ 4178us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:57.134118 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:57.137031 (+ 2913us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":375,"threads_started":1}
W20260928 08:44:57.137490 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:44:57.125128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:54073 (local address 127.0.97.62:42245)
0928 08:44:57.125288 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:57.125292 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:57.125709 (+ 417us) server_negotiation.cc:407] Connection header received
0928 08:44:57.125843 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:57.125856 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:57.125914 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:57.125993 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:57.126876 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:57.127690 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:57.128972 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:57.129561 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:57.129685 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:57.137234 (+ 7549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:54073: BlockingRecv error: recv got EOF from 127.0.97.2:54073 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 08:44:57 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:44:57 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:44:57.595404 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:57.586982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.3:56651)
0928 08:44:57.587512 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:57.587538 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:57.587650 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:57.587971 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:57.587974 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:57.588347 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:57.588493 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:57.588501 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:57.589635 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:57.589639 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:57.590524 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:57.590541 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:57.591319 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:57.591324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:57.591444 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:57.592068 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:57.592085 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:57.595017 (+ 2932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":303,"threads_started":1}
W20260928 08:44:57.595521 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:44:57.587232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56651 (local address 127.0.97.62:42245)
0928 08:44:57.587488 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:57.587493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:57.587620 (+ 127us) server_negotiation.cc:407] Connection header received
0928 08:44:57.587738 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:57.587742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:57.587829 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:57.587931 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:57.588636 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:57.589474 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:57.590715 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:57.591137 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:57.591238 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:57.595230 (+ 3992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56651: BlockingRecv error: recv got EOF from 127.0.97.3:56651 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 28 08:44:57 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:44:57 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:44:57.699215 9654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:57.690941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.1:51813)
0928 08:44:57.691445 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:57.691479 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:57.691611 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:57.692000 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:57.692004 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:57.692470 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:57.692664 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:57.692672 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:57.693652 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:57.693656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:57.694541 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:57.694557 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:57.695335 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:57.695339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:57.695512 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:57.696102 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:57.696118 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:57.698973 (+ 2855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":250,"threads_started":1}
W20260928 08:44:57.699503 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:44:57.691130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:51813 (local address 127.0.97.62:42245)
0928 08:44:57.691339 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:57.691359 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:57.691571 (+ 212us) server_negotiation.cc:407] Connection header received
0928 08:44:57.691727 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:57.691731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:57.691788 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:57.691937 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:57.692790 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:57.693512 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:57.694709 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:57.695200 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:57.695318 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:57.699179 (+ 3861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:51813: BlockingRecv error: recv got EOF from 127.0.97.1:51813 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 28 08:44:58 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:44:58 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:44:58.146564 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:58.138202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.2:51299)
0928 08:44:58.138876 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:58.138891 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:58.139044 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:58.139396 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:58.139401 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:58.139752 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:58.139966 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:58.139975 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:58.141102 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:58.141107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:58.141999 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:58.142009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:58.142689 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:58.142694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:58.142883 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:58.143429 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:58.143447 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:58.146321 (+ 2874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":263,"threads_started":1}
W20260928 08:44:58.146745 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:44:58.138314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:51299 (local address 127.0.97.62:42245)
0928 08:44:58.138578 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:58.138584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:58.138978 (+ 394us) server_negotiation.cc:407] Connection header received
0928 08:44:58.139183 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:58.139189 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:58.139258 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:58.139372 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:58.140098 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:58.140966 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:58.142138 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:58.142545 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:58.142695 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:58.146499 (+ 3804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:51299: BlockingRecv error: recv got EOF from 127.0.97.2:51299 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
Sep 28 08:44:58 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:44:58 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:44:58.605315 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:58.596349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.3:40359)
0928 08:44:58.596923 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:58.596942 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:58.597126 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:58.597499 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:58.597504 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:58.597895 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:58.598070 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:58.598079 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:58.599111 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:58.599120 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:58.600268 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:58.600281 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:58.601065 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:58.601072 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:58.601259 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:58.601904 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:58.601963 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:58.604957 (+ 2994us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":306,"threads_started":1}
W20260928 08:44:58.605451 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:44:58.596516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40359 (local address 127.0.97.62:42245)
0928 08:44:58.596762 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:58.596770 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:58.597054 (+ 284us) server_negotiation.cc:407] Connection header received
0928 08:44:58.597258 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:58.597263 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:58.597355 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:58.597467 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:58.598224 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:58.598978 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:58.600472 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:58.600912 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:58.601060 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:58.605218 (+ 4158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40359: BlockingRecv error: recv got EOF from 127.0.97.3:40359 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 28 08:44:58 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:44:58 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:44:58.709144 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:58.700389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.1:42773)
0928 08:44:58.701210 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:58.701232 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:58.701388 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:58.701795 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:58.701800 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:58.702167 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:58.702325 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:58.702333 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:58.703291 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:58.703295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:58.704383 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:58.704393 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:58.705089 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:58.705093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:58.705221 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:58.705837 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:58.705859 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:58.708900 (+ 3041us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":541,"threads_started":1}
W20260928 08:44:58.709369 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:44:58.700526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42773 (local address 127.0.97.62:42245)
0928 08:44:58.700768 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:58.700773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:58.701323 (+ 550us) server_negotiation.cc:407] Connection header received
0928 08:44:58.701528 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:58.701535 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:58.701624 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:58.701732 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:58.702493 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:58.703177 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:58.704565 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:58.704962 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:58.705067 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:58.709107 (+ 4040us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42773: BlockingRecv error: recv got EOF from 127.0.97.1:42773 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 08:44:59 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:44:59 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:44:59.155754 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:59.147552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.2:55041)
0928 08:44:59.148093 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:59.148108 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:59.148265 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:59.148639 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:59.148644 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:59.149018 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:59.149157 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:59.149164 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:59.150129 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:59.150138 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:59.151085 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:59.151095 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:59.151801 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:59.151839 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:59.152028 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:59.152691 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:59.152708 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:59.155540 (+ 2832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":320,"threads_started":1}
W20260928 08:44:59.155992 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:44:59.147594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55041 (local address 127.0.97.62:42245)
0928 08:44:59.147877 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:59.147885 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:59.148198 (+ 313us) server_negotiation.cc:407] Connection header received
0928 08:44:59.148423 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:59.148429 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:59.148498 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:59.148610 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:59.149300 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:59.150002 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:59.151225 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:59.151646 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:59.151779 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:59.155715 (+ 3936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:55041: BlockingRecv error: recv got EOF from 127.0.97.2:55041 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 28 08:44:59 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:44:59 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:44:59.614982 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:59.606245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.3:36035)
0928 08:44:59.606716 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:59.606732 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:59.606842 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:59.607120 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:59.607123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:59.607467 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:59.607606 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:59.607613 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:59.609102 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:59.609111 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:59.610062 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:59.610073 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:59.610800 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:59.610814 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:59.610984 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:59.611636 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:59.611652 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:59.614628 (+ 2976us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":249,"threads_started":1}
W20260928 08:44:59.615113 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:44:59.606306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36035 (local address 127.0.97.62:42245)
0928 08:44:59.606525 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:59.606532 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:59.606814 (+ 282us) server_negotiation.cc:407] Connection header received
0928 08:44:59.606937 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:59.606940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:59.606997 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:59.607086 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:59.607745 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:59.608937 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:59.610233 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:59.610654 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:59.610787 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:59.614851 (+ 4064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:36035: BlockingRecv error: recv got EOF from 127.0.97.3:36035 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 08:44:59 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:44:59 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:44:59.718614 9660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:44:59.710113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.1:38591)
0928 08:44:59.710647 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0928 08:44:59.710664 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:44:59.710771 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:44:59.711080 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:44:59.711083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:44:59.711486 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:44:59.711644 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:59.711653 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:59.712622 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:59.712625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:59.713563 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:44:59.713575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:44:59.714245 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:44:59.714250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:44:59.714398 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:59.715122 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:44:59.715154 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:44:59.718345 (+ 3191us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":284,"threads_started":1}
W20260928 08:44:59.718781 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:44:59.710235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:38591 (local address 127.0.97.62:42245)
0928 08:44:59.710447 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 08:44:59.710451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:44:59.710740 (+ 289us) server_negotiation.cc:407] Connection header received
0928 08:44:59.710894 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:44:59.710897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:44:59.710955 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:44:59.711035 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:44:59.711778 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:59.712496 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:59.713719 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:44:59.714115 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:44:59.714215 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:44:59.718553 (+ 4338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:38591: BlockingRecv error: recv got EOF from 127.0.97.1:38591 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 08:45:00 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:45:00 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:45:00.167369 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:00.156797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.2:48387)
0928 08:45:00.157558 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:00.157575 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:00.157689 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:00.158098 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:00.158102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:00.158531 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:00.158697 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:00.158706 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:00.160441 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:00.160445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:00.161615 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:00.161629 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:00.162945 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:00.162952 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:00.163174 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:00.164065 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:45:00.164084 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:00.167122 (+ 3038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":498,"threads_started":1}
W20260928 08:45:00.167966 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:45:00.157054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48387 (local address 127.0.97.62:42245)
0928 08:45:00.157304 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:00.157313 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:00.157658 (+ 345us) server_negotiation.cc:407] Connection header received
0928 08:45:00.157833 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:00.157841 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:00.157939 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:00.158065 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:00.158862 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:00.160273 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:00.161798 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:00.162706 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:00.162863 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:00.167406 (+ 4543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:48387: BlockingRecv error: recv got EOF from 127.0.97.2:48387 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 28 08:45:00 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:45:00 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:45:00.624540 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:00.616087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.3:58325)
0928 08:45:00.616673 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:00.616689 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:00.616796 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:00.617083 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:00.617086 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:00.617496 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:00.617662 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:00.617670 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:00.618665 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:00.618669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:00.619732 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:00.619745 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:00.620515 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:00.620520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:00.620659 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:00.621249 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:45:00.621266 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:00.624303 (+ 3037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":329,"threads_started":1}
W20260928 08:45:00.624789 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:45:00.616141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58325 (local address 127.0.97.62:42245)
0928 08:45:00.616329 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:00.616335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:00.616746 (+ 411us) server_negotiation.cc:407] Connection header received
0928 08:45:00.616900 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:00.616904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:00.616962 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:00.617036 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:00.617787 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:00.618505 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:00.619988 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:00.620373 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:00.620463 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:00.624497 (+ 4034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58325: BlockingRecv error: recv got EOF from 127.0.97.3:58325 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 08:45:00 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:45:00 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:45:00.728335 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:00.719546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.1:56789)
0928 08:45:00.720180 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:00.720196 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:00.720318 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:00.720621 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:00.720624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:00.720972 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:00.721128 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:00.721136 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:00.722180 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:00.722185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:00.723735 (+ 1550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:00.723752 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:00.724506 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:00.724510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:00.724638 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:00.725287 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:45:00.725310 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:00.728051 (+ 2741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":414,"threads_started":1}
W20260928 08:45:00.728433 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:45:00.719757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56789 (local address 127.0.97.62:42245)
0928 08:45:00.719998 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:00.720006 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:00.720291 (+ 285us) server_negotiation.cc:407] Connection header received
0928 08:45:00.720415 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:00.720419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:00.720478 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:00.720551 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:00.721252 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:00.722008 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:00.723916 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:00.724356 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:00.724456 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:00.728226 (+ 3770us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:56789: BlockingRecv error: recv got EOF from 127.0.97.1:56789 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 08:45:01 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:45:01 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:45:01.177701 9664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:01.168397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.2:58253)
0928 08:45:01.168917 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:01.168935 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:01.169056 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:01.169387 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:01.169391 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:01.169815 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:01.170042 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:01.170051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:01.171103 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:01.171107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:01.172325 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:01.172337 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:01.173177 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:01.173181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:01.173358 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:01.173943 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:45:01.173961 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:01.177328 (+ 3367us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":293,"threads_started":1}
W20260928 08:45:01.177860 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:45:01.168503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:58253 (local address 127.0.97.62:42245)
0928 08:45:01.168782 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:01.168788 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:01.169103 (+ 315us) server_negotiation.cc:407] Connection header received
0928 08:45:01.169164 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:01.169167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:01.169230 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:01.169322 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:01.170182 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:01.170972 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:01.172579 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:01.173024 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:01.173130 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:01.177546 (+ 4416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:58253: BlockingRecv error: recv got EOF from 127.0.97.2:58253 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Sep 28 08:45:01 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:45:01 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:45:01.634897 9665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:01.625542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.3:59251)
0928 08:45:01.626094 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:01.626112 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:01.626236 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:01.626608 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:01.626612 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:01.626965 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:01.627170 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:01.627182 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:01.628757 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:01.628763 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:01.629892 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:01.629905 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:01.630647 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:01.630652 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:01.630795 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:01.631659 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:45:01.631677 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:01.634678 (+ 3001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":318,"threads_started":1}
W20260928 08:45:01.635124 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:45:01.625652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59251 (local address 127.0.97.62:42245)
0928 08:45:01.625878 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:01.625883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:01.626206 (+ 323us) server_negotiation.cc:407] Connection header received
0928 08:45:01.626396 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:01.626400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:01.626461 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:01.626555 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:01.627387 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:01.628561 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:01.630039 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:01.630503 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:01.630598 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:01.634876 (+ 4278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:59251: BlockingRecv error: recv got EOF from 127.0.97.3:59251 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 28 08:45:01 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:45:01 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:45:01.738238 9666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:01.729530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.1:48039)
0928 08:45:01.730150 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:01.730167 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:01.730303 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:01.730631 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:01.730635 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:01.731057 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:01.731221 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:01.731229 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:01.732324 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:01.732330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:01.733283 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:01.733296 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:01.734033 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:01.734037 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:01.734162 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:01.734768 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:45:01.734784 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:01.737926 (+ 3142us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":311,"threads_started":1}
W20260928 08:45:01.738447 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:45:01.729648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48039 (local address 127.0.97.62:42245)
0928 08:45:01.729970 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:01.729977 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:01.730322 (+ 345us) server_negotiation.cc:407] Connection header received
0928 08:45:01.730419 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:01.730423 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:01.730486 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:01.730578 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:01.731370 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:01.732183 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:01.733468 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:01.733874 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:01.733987 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:01.738177 (+ 4190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48039: BlockingRecv error: recv got EOF from 127.0.97.1:48039 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Sep 28 08:45:02 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:45:02 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:45:02.194414 9667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:02.179020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.2:40653)
0928 08:45:02.179739 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:02.179786 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:02.179979 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:02.180414 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:02.180419 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:02.181041 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:02.181246 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:02.181258 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:02.182442 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:02.182447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:02.183433 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:02.183446 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:02.184377 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:02.184385 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:02.184572 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:02.190507 (+ 5935us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:45:02.190542 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:02.194144 (+ 3602us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":369,"threads_started":1}
W20260928 08:45:02.194635 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:45:02.179231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40653 (local address 127.0.97.62:42245)
0928 08:45:02.179386 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:02.179391 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:02.179912 (+ 521us) server_negotiation.cc:407] Connection header received
0928 08:45:02.180118 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:02.180123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:02.180235 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:02.180329 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:02.181420 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:02.182287 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:02.183565 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:02.184173 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:02.184280 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:02.194339 (+ 10059us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:40653: BlockingRecv error: recv got EOF from 127.0.97.2:40653 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 28 08:45:02 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:45:02 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:45:02.644366 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:02.635951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.3:58667)
0928 08:45:02.636563 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:02.636578 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:02.636687 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:02.636985 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:02.636988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:02.637423 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:02.637583 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:02.637591 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:02.638611 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:02.638615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:02.639557 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:02.639568 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:02.640523 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:02.640527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:02.640636 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:02.641311 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:45:02.641328 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:02.644129 (+ 2801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":265,"threads_started":1}
W20260928 08:45:02.644795 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:45:02.635981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58667 (local address 127.0.97.62:42245)
0928 08:45:02.636195 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:02.636201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:02.636661 (+ 460us) server_negotiation.cc:407] Connection header received
0928 08:45:02.636792 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:02.636796 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:02.636854 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:02.636934 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:02.637723 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:02.638467 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:02.639829 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:02.640371 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:02.640492 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:02.644374 (+ 3882us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58667: BlockingRecv error: recv got EOF from 127.0.97.3:58667 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 28 08:45:02 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:45:02 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:45:02.748369 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:02.739235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.1:40573)
0928 08:45:02.739900 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:02.739928 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:02.740049 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:02.740413 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:02.740417 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:02.740903 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:02.741052 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:02.741060 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:02.742164 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:02.742168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:02.743090 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:02.743100 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:02.743788 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:02.743793 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:02.743965 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:02.744599 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:45:02.744617 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:02.748037 (+ 3420us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":324,"threads_started":1}
W20260928 08:45:02.748558 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:45:02.739587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40573 (local address 127.0.97.62:42245)
0928 08:45:02.739854 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:02.739885 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:02.740044 (+ 159us) server_negotiation.cc:407] Connection header received
0928 08:45:02.740140 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:02.740144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:02.740206 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:02.740323 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:02.741183 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:02.742025 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:02.743225 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:02.743627 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:02.743722 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:02.748284 (+ 4562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:40573: BlockingRecv error: recv got EOF from 127.0.97.1:40573 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 28 08:45:03 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:45:03 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:45:03.204350 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:03.195512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.2:43803)
0928 08:45:03.196106 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:03.196121 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:03.196340 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:03.196694 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:03.196698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:03.197179 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:03.197392 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:03.197403 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:03.198580 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:03.198586 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:03.199491 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:03.199502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:03.200289 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:03.200295 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:03.200450 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:03.201103 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:45:03.201121 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:03.204099 (+ 2978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":318,"threads_started":1}
W20260928 08:45:03.204507 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:45:03.195677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:43803 (local address 127.0.97.62:42245)
0928 08:45:03.195969 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:03.195976 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:03.196210 (+ 234us) server_negotiation.cc:407] Connection header received
0928 08:45:03.196481 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:03.196489 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:03.196557 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:03.196668 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:03.197549 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:03.198417 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:03.199636 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:03.200095 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:03.200240 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:03.204293 (+ 4053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:43803: BlockingRecv error: recv got EOF from 127.0.97.2:43803 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 28 08:45:03 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:45:03 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:45:03.653777 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:03.645275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.3:40007)
0928 08:45:03.645805 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:03.645821 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:03.645931 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:03.646217 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:03.646220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:03.646637 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:03.646783 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:03.646791 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:03.647857 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:03.647861 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:03.648769 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:03.648779 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:03.649500 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:03.649505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:03.649610 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:03.650166 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:45:03.650182 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:03.653417 (+ 3235us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":376,"threads_started":1}
W20260928 08:45:03.654049 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:45:03.645464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40007 (local address 127.0.97.62:42245)
0928 08:45:03.645675 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:03.645679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:03.645904 (+ 225us) server_negotiation.cc:407] Connection header received
0928 08:45:03.646032 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:03.646035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:03.646094 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:03.646172 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:03.646910 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:03.647669 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:03.648924 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:03.649333 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:03.649433 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:03.653679 (+ 4246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:40007: BlockingRecv error: recv got EOF from 127.0.97.3:40007 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 28 08:45:03 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:45:03 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:45:03.758486 9672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:03.749375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.1:37043)
0928 08:45:03.750170 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:03.750191 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:03.750396 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:03.750836 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:03.750842 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:03.751437 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:03.751608 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:03.751617 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:03.752876 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:03.752885 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:03.753952 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:03.753964 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:03.754644 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:03.754649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:03.754797 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:03.755391 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:45:03.755408 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:03.758233 (+ 2825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":518,"threads_started":1}
W20260928 08:45:03.758735 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:45:03.749590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37043 (local address 127.0.97.62:42245)
0928 08:45:03.749799 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:03.749806 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:03.750287 (+ 481us) server_negotiation.cc:407] Connection header received
0928 08:45:03.750552 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:03.750559 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:03.750646 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:03.750743 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:03.751768 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:03.752750 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:03.754073 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:03.754483 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:03.754587 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:03.758472 (+ 3885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:37043: BlockingRecv error: recv got EOF from 127.0.97.1:37043 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 08:45:04 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:45:04 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:45:04.213999 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:04.205235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.2:46165)
0928 08:45:04.205811 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:04.205830 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:04.205953 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:04.206345 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:04.206349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:04.206751 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:04.206948 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:04.206957 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:04.208130 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:04.208136 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:04.209255 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:04.209268 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:04.209991 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:04.209996 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:04.210127 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:04.210756 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:45:04.210773 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:04.213656 (+ 2883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":307,"threads_started":1}
W20260928 08:45:04.214139 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:45:04.205351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46165 (local address 127.0.97.62:42245)
0928 08:45:04.205510 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:04.205515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:04.205943 (+ 428us) server_negotiation.cc:407] Connection header received
0928 08:45:04.206108 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:04.206112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:04.206174 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:04.206292 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:04.207097 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:04.207990 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:04.209389 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:04.209817 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:04.209917 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:04.213870 (+ 3953us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:46165: BlockingRecv error: recv got EOF from 127.0.97.2:46165 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 08:45:04 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:45:04 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:45:04.664021 9674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:04.654996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.3:57481)
0928 08:45:04.655566 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:04.655582 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:04.655753 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:04.656335 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:04.656339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:04.656847 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:04.657088 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:04.657101 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:04.658275 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:04.658279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:04.659301 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:04.659314 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:04.660053 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:04.660058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:04.660208 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:04.660772 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:45:04.660791 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:04.663736 (+ 2945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":324,"threads_started":1}
W20260928 08:45:04.664166 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:45:04.655073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57481 (local address 127.0.97.62:42245)
0928 08:45:04.655311 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:04.655318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:04.655677 (+ 359us) server_negotiation.cc:407] Connection header received
0928 08:45:04.655954 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:04.655958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:04.656020 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:04.656320 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:04.657291 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:04.658113 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:04.659435 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:04.659909 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:04.660002 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:04.663983 (+ 3981us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:57481: BlockingRecv error: recv got EOF from 127.0.97.3:57481 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 08:45:04 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:45:04 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:45:04.768115 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:04.759465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.1:53519)
0928 08:45:04.760109 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:04.760125 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:04.760248 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:04.760576 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:04.760579 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:04.760948 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:04.761094 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:04.761102 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:04.762344 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:04.762348 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:04.763287 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:04.763298 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:04.764195 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:04.764202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:04.764365 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:04.765038 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:45:04.765071 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:04.767889 (+ 2818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":431,"threads_started":1}
W20260928 08:45:04.768275 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:45:04.759611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53519 (local address 127.0.97.62:42245)
0928 08:45:04.759882 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:04.759887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:04.760235 (+ 348us) server_negotiation.cc:407] Connection header received
0928 08:45:04.760368 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:04.760372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:04.760434 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:04.760521 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:04.761249 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:04.762163 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:04.763440 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:04.764036 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:04.764133 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:04.768028 (+ 3895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53519: BlockingRecv error: recv got EOF from 127.0.97.1:53519 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 28 08:45:05 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:45:05 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:45:05.223362 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:05.214941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.2:60961)
0928 08:45:05.215457 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:05.215473 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:05.215580 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:05.215925 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:05.215928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:05.216256 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:05.216412 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:05.216420 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:05.217477 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:05.217481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:05.218393 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:05.218404 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:05.219098 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:05.219102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:05.219220 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:05.220047 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:45:05.220065 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:05.223135 (+ 3070us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":300,"threads_started":1}
W20260928 08:45:05.223572 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:45:05.215043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60961 (local address 127.0.97.62:42245)
0928 08:45:05.215227 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:05.215231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:05.215577 (+ 346us) server_negotiation.cc:407] Connection header received
0928 08:45:05.215670 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:05.215674 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:05.215738 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:05.215872 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:05.216542 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:05.217353 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:05.218537 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:05.218970 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:05.219073 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:05.223346 (+ 4273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:60961: BlockingRecv error: recv got EOF from 127.0.97.2:60961 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 08:45:05 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:45:05 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:45:05.673815 9677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:05.665130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.3:59067)
0928 08:45:05.665721 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:05.665738 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:05.665840 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:05.666240 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:05.666243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:05.666598 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:05.666747 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:05.666755 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:05.667939 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:05.667943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:05.668938 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:05.668950 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:05.669698 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:05.669702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:05.669862 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:05.670451 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:45:05.670467 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:05.673571 (+ 3104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":386,"threads_started":1}
W20260928 08:45:05.673986 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:45:05.665265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59067 (local address 127.0.97.62:42245)
0928 08:45:05.665631 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:05.665636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:05.665809 (+ 173us) server_negotiation.cc:407] Connection header received
0928 08:45:05.665980 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:05.665984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:05.666049 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:05.666139 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:05.666912 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:05.667753 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:05.669082 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:05.669561 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:05.669659 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:05.673773 (+ 4114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:59067: BlockingRecv error: recv got EOF from 127.0.97.3:59067 (error 108)
Metrics: {"server-negotiator.queue_time_us":241}
Sep 28 08:45:05 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:45:05 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:45:05.777911 9678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:05.769042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.1:42269)
0928 08:45:05.769580 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:05.769596 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:05.769756 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:05.770122 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:05.770129 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:05.770558 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:05.770730 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:05.770739 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:05.771856 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:05.771864 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:05.772952 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:05.772963 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:05.773654 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:05.773660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:05.773834 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:05.774413 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:45:05.774430 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:05.777545 (+ 3115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":306,"threads_started":1}
W20260928 08:45:05.777981 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:45:05.769140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:42269 (local address 127.0.97.62:42245)
0928 08:45:05.769455 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:05.769461 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:05.769672 (+ 211us) server_negotiation.cc:407] Connection header received
0928 08:45:05.769891 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:05.769900 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:05.769984 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:05.770085 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:05.770886 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:05.771715 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:05.773092 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:05.773510 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:05.773639 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:05.777754 (+ 4115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:42269: BlockingRecv error: recv got EOF from 127.0.97.1:42269 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
Sep 28 08:45:06 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:45:06 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:45:06.232990 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:06.224345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.2:52019)
0928 08:45:06.224881 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:06.224898 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:06.225061 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:06.225617 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:06.225622 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:06.226057 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:06.226238 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:06.226246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:06.227276 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:06.227283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:06.228247 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:06.228258 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:06.228956 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:06.228961 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:06.229140 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:06.229737 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:45:06.229753 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:06.232733 (+ 2980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":284,"threads_started":1}
W20260928 08:45:06.233140 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:45:06.224451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52019 (local address 127.0.97.62:42245)
0928 08:45:06.224686 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:06.224695 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:06.224982 (+ 287us) server_negotiation.cc:407] Connection header received
0928 08:45:06.225327 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:06.225331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:06.225396 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:06.225590 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:06.226395 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:06.227137 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:06.228390 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:06.228792 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:06.228928 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:06.232918 (+ 3990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:52019: BlockingRecv error: recv got EOF from 127.0.97.2:52019 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 28 08:45:06 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:45:06 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:45:06.683986 9680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:06.674866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.3:56781)
0928 08:45:06.675475 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:06.675491 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:06.675601 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:06.675946 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:06.675950 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:06.676337 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:06.676521 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:06.676529 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:06.677527 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:06.677531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:06.678792 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:06.678806 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:06.679894 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:06.679900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:06.680036 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:06.680640 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:45:06.680658 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:06.683600 (+ 2942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":375,"threads_started":1}
W20260928 08:45:06.684036 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:45:06.675060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:56781 (local address 127.0.97.62:42245)
0928 08:45:06.675228 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:06.675232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:06.675562 (+ 330us) server_negotiation.cc:407] Connection header received
0928 08:45:06.675716 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:06.675719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:06.675776 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:06.675870 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:06.676684 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:06.677402 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:06.678940 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:06.679683 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:06.679864 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:06.683793 (+ 3929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:56781: BlockingRecv error: recv got EOF from 127.0.97.3:56781 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 08:45:06 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:45:06 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:45:06.787845 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:06.778868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.1:53063)
0928 08:45:06.779352 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:06.779367 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:06.779478 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:06.779864 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:06.779870 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:06.780365 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:06.780520 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:06.780534 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:06.781639 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:06.781646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:06.782787 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:06.782798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:06.783505 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:06.783510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:06.783643 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:06.784300 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:45:06.784323 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:06.787482 (+ 3159us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":270,"threads_started":1}
W20260928 08:45:06.787925 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:45:06.778913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:53063 (local address 127.0.97.62:42245)
0928 08:45:06.779156 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:06.779163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:06.779448 (+ 285us) server_negotiation.cc:407] Connection header received
0928 08:45:06.779589 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:06.779593 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:06.779651 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:06.779730 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:06.780655 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:06.781435 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:06.782921 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:06.783356 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:06.783453 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:06.787656 (+ 4203us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:53063: BlockingRecv error: recv got EOF from 127.0.97.1:53063 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 28 08:45:07 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:45:07 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:45:07.253069 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:07.234132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.2:52311)
0928 08:45:07.234645 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:07.234662 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:07.234779 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:07.235128 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:07.235131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:07.235519 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:07.235664 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:07.235673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:07.236974 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:07.236979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:07.238086 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:07.238097 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:07.238792 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:07.238795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:07.238907 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:07.249402 (+ 10495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:45:07.249429 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:07.252707 (+ 3278us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":284,"threads_started":1}
W20260928 08:45:07.253190 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:45:07.234211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:52311 (local address 127.0.97.62:42245)
0928 08:45:07.234472 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:07.234480 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:07.234763 (+ 283us) server_negotiation.cc:407] Connection header received
0928 08:45:07.234904 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:07.234908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:07.234974 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:07.235093 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:07.235784 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:07.236794 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:07.238227 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:07.238665 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:07.238774 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:07.252919 (+ 14145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:52311: BlockingRecv error: recv got EOF from 127.0.97.2:52311 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 28 08:45:07 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:45:07 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:45:07.693485 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:07.685201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.3:43803)
0928 08:45:07.685780 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:07.685798 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:07.685931 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:07.686295 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:07.686298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:07.686684 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:07.686834 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:07.686841 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:07.687934 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:07.687938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:07.688911 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:07.688921 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:07.689594 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:07.689598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:07.689723 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:07.690353 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:45:07.690369 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:07.693265 (+ 2896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":309,"threads_started":1}
W20260928 08:45:07.693727 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:45:07.685318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43803 (local address 127.0.97.62:42245)
0928 08:45:07.685474 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:07.685478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:07.685901 (+ 423us) server_negotiation.cc:407] Connection header received
0928 08:45:07.686027 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:07.686030 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:07.686106 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:07.686232 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:07.686973 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:07.687739 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:07.689044 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:07.689461 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:07.689551 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:07.693466 (+ 3915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:43803: BlockingRecv error: recv got EOF from 127.0.97.3:43803 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 08:45:07 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:45:07 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:45:07.797277 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:07.788694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.1:54361)
0928 08:45:07.789266 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:07.789283 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:07.789395 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:07.789746 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:07.789749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:07.790105 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:07.790281 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:07.790290 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:07.791455 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:07.791460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:07.792725 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:07.792737 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:07.793400 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:07.793405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:07.793515 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:07.794144 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:45:07.794161 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:07.797026 (+ 2865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":280,"threads_started":1}
W20260928 08:45:07.797438 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:45:07.788811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:54361 (local address 127.0.97.62:42245)
0928 08:45:07.789000 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:07.789004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:07.789364 (+ 360us) server_negotiation.cc:407] Connection header received
0928 08:45:07.789543 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:07.789548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:07.789609 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:07.789704 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:07.790420 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:07.791303 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:07.792846 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:07.793252 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:07.793344 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:07.797206 (+ 3862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:54361: BlockingRecv error: recv got EOF from 127.0.97.1:54361 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 28 08:45:08 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:45:08 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:45:08.262603 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:08.254164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.2:47861)
0928 08:45:08.254865 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:08.254880 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:08.255057 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:08.255469 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:08.255476 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:08.255920 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:08.256100 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:08.256108 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:08.257093 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:08.257097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:08.258012 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:08.258022 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:08.258848 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:08.258852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:08.258964 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:08.259566 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:45:08.259581 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:08.262366 (+ 2785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":424,"threads_started":1}
W20260928 08:45:08.262840 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:45:08.254346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:47861 (local address 127.0.97.62:42245)
0928 08:45:08.254574 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:08.254583 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:08.254985 (+ 402us) server_negotiation.cc:407] Connection header received
0928 08:45:08.255230 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:08.255235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:08.255312 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:08.255438 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:08.256230 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:08.256944 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:08.258206 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:08.258685 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:08.258791 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:08.262569 (+ 3778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:47861: BlockingRecv error: recv got EOF from 127.0.97.2:47861 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 28 08:45:08 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:45:08 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:45:08.702744 9686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:08.694496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.3:38883)
0928 08:45:08.695224 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:08.695239 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:08.695387 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:08.695736 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:08.695741 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:08.696218 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:08.696383 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:08.696391 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:08.697353 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:08.697359 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:08.698249 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:08.698259 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:08.698962 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:08.698968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:08.699128 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:08.699736 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:45:08.699752 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:08.702415 (+ 2663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":443,"threads_started":1}
W20260928 08:45:08.702829 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:45:08.694549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38883 (local address 127.0.97.62:42245)
0928 08:45:08.694757 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:08.694767 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:08.695329 (+ 562us) server_negotiation.cc:407] Connection header received
0928 08:45:08.695519 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:08.695525 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:08.695597 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:08.695709 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:08.696518 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:08.697222 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:08.698380 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:08.698791 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:08.698922 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:08.702604 (+ 3682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:38883: BlockingRecv error: recv got EOF from 127.0.97.3:38883 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 08:45:08 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:45:08 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:45:08.806710 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:08.798228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.1:58607)
0928 08:45:08.798771 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:08.798787 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:08.798937 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:08.799350 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:08.799358 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:08.799757 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:08.799954 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:08.799963 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:08.801175 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:08.801184 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:08.802366 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:08.802378 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:08.803059 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:08.803064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:08.803258 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:08.803853 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:45:08.803869 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:08.806485 (+ 2616us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":269,"threads_started":1}
W20260928 08:45:08.807011 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:45:08.798309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:58607 (local address 127.0.97.62:42245)
0928 08:45:08.798639 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:08.798645 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:08.798874 (+ 229us) server_negotiation.cc:407] Connection header received
0928 08:45:08.799082 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:08.799087 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:08.799200 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:08.799311 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:08.800113 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:08.801022 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:08.802499 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:08.802920 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:08.803050 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:08.806653 (+ 3603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:58607: BlockingRecv error: recv got EOF from 127.0.97.1:58607 (error 108)
Metrics: {"server-negotiator.queue_time_us":155}
Sep 28 08:45:09 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:45:09 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:45:09.271617 9688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:09.263565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.2:38927)
0928 08:45:09.264078 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:09.264094 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:09.264213 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:09.264518 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:09.264521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:09.264877 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:09.265019 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:09.265026 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:09.266020 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:09.266024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:09.266955 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:09.266966 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:09.267688 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:09.267692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:09.267870 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:09.268564 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:45:09.268580 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:09.271326 (+ 2746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":286,"threads_started":1}
W20260928 08:45:09.271888 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:45:09.263676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38927 (local address 127.0.97.62:42245)
0928 08:45:09.263895 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:09.263903 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:09.264188 (+ 285us) server_negotiation.cc:407] Connection header received
0928 08:45:09.264334 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:09.264337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:09.264398 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:09.264478 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:09.265172 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:09.265885 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:09.267111 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:09.267543 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:09.267640 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:09.271598 (+ 3958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:38927: BlockingRecv error: recv got EOF from 127.0.97.2:38927 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 08:45:09 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:45:09 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.3@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:45:09.711428 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:09.703633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.3:58893)
0928 08:45:09.704126 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:09.704142 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:09.704251 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:09.704574 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:09.704577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:09.704925 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:09.705089 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:09.705097 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:09.706079 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:09.706083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:09.706984 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:09.706994 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:09.707690 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:09.707694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:09.707871 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:09.708483 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:45:09.708502 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:09.711209 (+ 2707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":298,"threads_started":1}
W20260928 08:45:09.711676 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:45:09.703703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:58893 (local address 127.0.97.62:42245)
0928 08:45:09.703965 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:09.703972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:09.704236 (+ 264us) server_negotiation.cc:407] Connection header received
0928 08:45:09.704350 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:09.704353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:09.704427 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:09.704518 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:09.705237 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:09.705946 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:09.707146 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:09.707570 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:09.707659 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:09.711402 (+ 3743us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.3:58893: BlockingRecv error: recv got EOF from 127.0.97.3:58893 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 28 08:45:09 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:45:09 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.1@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:45:09.816557 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:09.807891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.1:48871)
0928 08:45:09.808408 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:09.808423 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:09.808609 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:09.809020 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:09.809024 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:09.809657 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:09.809864 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:09.809872 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:09.810913 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:09.810920 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:09.811898 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:09.811909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:09.812640 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:09.812646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:09.812818 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:09.813423 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:45:09.813445 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:09.816184 (+ 2739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":293,"threads_started":1}
W20260928 08:45:09.816613 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:45:09.807984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48871 (local address 127.0.97.62:42245)
0928 08:45:09.808222 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:09.808231 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:09.808530 (+ 299us) server_negotiation.cc:407] Connection header received
0928 08:45:09.808754 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:09.808760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:09.808858 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:09.808976 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:09.810015 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:09.810775 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:09.812026 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:09.812467 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:09.812621 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:09.816362 (+ 3741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.1:48871: BlockingRecv error: recv got EOF from 127.0.97.1:48871 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 08:45:10 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 08:45:10 dist-test-slave-n6xs krb5kdc[8645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.97.2@KRBTEST.COM for oryx/127.0.97.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 08:45:10.281821 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:10.273083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42245 (local address 127.0.97.2:41389)
0928 08:45:10.273683 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:10.273700 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:10.273809 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:10.274117 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:10.274121 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:10.274509 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:10.274684 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:10.274691 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:10.275776 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:10.275779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:10.276728 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:10.276738 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:10.277541 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:10.277547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:10.277729 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:10.278411 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:45:10.278455 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:10.281495 (+ 3040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.97.62:42245: Server oryx/127.0.97.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":335,"threads_started":1}
W20260928 08:45:10.281849 9383 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:45:10.273180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:41389 (local address 127.0.97.62:42245)
0928 08:45:10.273340 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:10.273344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:10.273779 (+ 435us) server_negotiation.cc:407] Connection header received
0928 08:45:10.273907 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:10.273911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:10.273986 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:10.274072 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:10.274811 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:10.275646 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:10.276840 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:10.277356 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:10.277490 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:10.281668 (+ 4178us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.97.2:41389: BlockingRecv error: recv got EOF from 127.0.97.2:41389 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 08:45:10.546308 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 9232
I20260928 08:45:10.558667 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:10.547345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43396 (local address 127.0.97.1:39403)
0928 08:45:10.547959 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:10.547966 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:10.547992 (+ 26us) server_negotiation.cc:407] Connection header received
0928 08:45:10.548048 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:10.548054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:10.548299 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:10.548452 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:10.549978 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:10.550993 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:10.552449 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:10.553012 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:10.553180 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:10.555689 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:45:10.555697 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:45:10.555712 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:45:10.555742 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:45:10.558204 (+ 2462us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 08:45:10.558444 (+ 240us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43396: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":301,"threads_started":1}
W20260928 08:45:10.558833 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 9232: Not authorized: Client connection negotiation failed: client connection to 127.0.97.1:39403: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 08:45:10.559049 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 9232
I20260928 08:45:10.575742 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 9384
I20260928 08:45:10.588322 9694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:10.576868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52982 (local address 127.0.97.2:39889)
0928 08:45:10.577483 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:10.577490 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:10.577520 (+ 30us) server_negotiation.cc:407] Connection header received
0928 08:45:10.577583 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:10.577590 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:10.577800 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:10.577957 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:10.579411 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:10.580594 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:10.581905 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:10.582452 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:10.582557 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:10.585075 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:45:10.585087 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:45:10.585107 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:45:10.585145 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:45:10.587724 (+ 2579us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 08:45:10.588055 (+ 331us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52982: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":325,"threads_started":1}
W20260928 08:45:10.588435 9694 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52982: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 08:45:10.588486 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 9384: Not authorized: Client connection negotiation failed: client connection to 127.0.97.2:39889: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 08:45:10.588686 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 9384
I20260928 08:45:10.605165 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 9515
I20260928 08:45:10.617728 9695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:10.606212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38596 (local address 127.0.97.3:34697)
0928 08:45:10.606958 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:10.606966 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:10.606996 (+ 30us) server_negotiation.cc:407] Connection header received
0928 08:45:10.607084 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:10.607091 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:10.607319 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:10.607491 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:10.608917 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:10.609982 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:10.611198 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:10.611742 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:10.611997 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:10.614449 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:45:10.614458 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:45:10.614473 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:45:10.614517 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:45:10.617223 (+ 2706us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 08:45:10.617492 (+ 269us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38596: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":447,"threads_started":1}
W20260928 08:45:10.617854 9695 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38596: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 08:45:10.617952 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 9515: Not authorized: Client connection negotiation failed: client connection to 127.0.97.3:34697: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.97.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 08:45:10.618139 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 9515
I20260928 08:45:10.634297 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 9163
I20260928 08:45:10.635581 9225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:45:10.750520 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 9163
2026-09-28T08:45:10Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21052 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T08:45:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:45:10Z Disabled control of system clock
I20260928 08:45:10.831431 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45881
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:42335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45881
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 08:45:11.158367 9700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:45:11.158699 9700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:45:11.158860 9700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:45:11.166209 9700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:45:11.166373 9700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:45:11.166466 9700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:45:11.166575 9700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:45:11.166651 9700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:45:11.176003 9700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45881
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:45:11.177336 9700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:45:11.179111 9700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:45:11.191412 9708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:45:11.191448 9706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:45:11.191308 9705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:45:11.193307 9700 server_base.cc:1034] running on GCE node
I20260928 08:45:11.194216 9700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:45:11.195740 9700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:45:11.197041 9700 hybrid_clock.cc:648] HybridClock initialized: now 1790585111196980 us; error 80 us; skew 500 ppm
I20260928 08:45:11.197494 9700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:45:11.200052 9700 webserver.cc:460] Webserver started at http://127.0.97.62:38573/ using document root <none> and password file <none>
I20260928 08:45:11.200814 9700 fs_manager.cc:359] Metadata directory not provided
I20260928 08:45:11.200958 9700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:45:11.201318 9700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:45:11.204561 9700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "00f35b86d30e401f9d790f52b43b6fc6"
format_stamp: "Formatted at 2026-09-28 08:45:11 on dist-test-slave-n6xs"
server_key: "a9e021337adda29896528b565150cb91"
server_key_iv: "9962aa843e4495b973ce1c852524997f"
server_key_version: "encryptionkey@0"
I20260928 08:45:11.205417 9700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "00f35b86d30e401f9d790f52b43b6fc6"
format_stamp: "Formatted at 2026-09-28 08:45:11 on dist-test-slave-n6xs"
server_key: "a9e021337adda29896528b565150cb91"
server_key_iv: "9962aa843e4495b973ce1c852524997f"
server_key_version: "encryptionkey@0"
I20260928 08:45:11.210887 9700 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 08:45:11.214679 9714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:45:11.216228 9700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:45:11.216421 9700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "00f35b86d30e401f9d790f52b43b6fc6"
format_stamp: "Formatted at 2026-09-28 08:45:11 on dist-test-slave-n6xs"
server_key: "a9e021337adda29896528b565150cb91"
server_key_iv: "9962aa843e4495b973ce1c852524997f"
server_key_version: "encryptionkey@0"
I20260928 08:45:11.216674 9700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:45:11.248965 9700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:45:11.249894 9700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:45:11.250227 9700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:45:11.270041 9700 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45881
I20260928 08:45:11.270049 9765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45881 every 8 connection(s)
I20260928 08:45:11.271471 9700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:45:11.273407 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 9700
I20260928 08:45:11.273669 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:45:11.274343 388 external_mini_cluster.cc:1428] Setting key 83ca0b1950f788b2bc78a17c7b7ae1bb
I20260928 08:45:11.275633 9766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:45:11.289881 9769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:11.276519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49870 (local address 127.0.97.62:45881)
0928 08:45:11.277578 (+ 1059us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:11.277592 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:11.277641 (+ 49us) server_negotiation.cc:407] Connection header received
0928 08:45:11.279199 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:11.279231 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:11.279865 (+ 634us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:11.280297 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:11.280851 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:11.282862 (+ 2011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:11.284604 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:11.285588 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:11.285791 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:11.286927 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:45:11.286937 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:45:11.286962 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:45:11.287745 (+ 783us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:45:11.287860 (+ 115us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:45:11.287886 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:45:11.288014 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 08:45:11.288222 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 08:45:11.288589 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":337,"threads_started":1}
I20260928 08:45:11.290747 9766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00f35b86d30e401f9d790f52b43b6fc6: Bootstrap starting.
I20260928 08:45:11.293929 9766 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00f35b86d30e401f9d790f52b43b6fc6: Neither blocks nor log segments found. Creating new log.
I20260928 08:45:11.294926 9766 log.cc:824] T 00000000000000000000000000000000 P 00f35b86d30e401f9d790f52b43b6fc6: Log is configured to *not* fsync() on all Append() calls
I20260928 08:45:11.297683 9766 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00f35b86d30e401f9d790f52b43b6fc6: No bootstrap required, opened a new log
I20260928 08:45:11.304867 9766 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00f35b86d30e401f9d790f52b43b6fc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f35b86d30e401f9d790f52b43b6fc6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45881 } }
I20260928 08:45:11.305270 9766 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00f35b86d30e401f9d790f52b43b6fc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:45:11.305377 9766 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00f35b86d30e401f9d790f52b43b6fc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00f35b86d30e401f9d790f52b43b6fc6, State: Initialized, Role: FOLLOWER
I20260928 08:45:11.305967 9766 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00f35b86d30e401f9d790f52b43b6fc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f35b86d30e401f9d790f52b43b6fc6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45881 } }
I20260928 08:45:11.306226 9766 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00f35b86d30e401f9d790f52b43b6fc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:45:11.306372 9766 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00f35b86d30e401f9d790f52b43b6fc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:45:11.306547 9766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00f35b86d30e401f9d790f52b43b6fc6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:45:11.308736 9766 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00f35b86d30e401f9d790f52b43b6fc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f35b86d30e401f9d790f52b43b6fc6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45881 } }
I20260928 08:45:11.309274 9766 leader_election.cc:302] T 00000000000000000000000000000000 P 00f35b86d30e401f9d790f52b43b6fc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00f35b86d30e401f9d790f52b43b6fc6; no voters:
I20260928 08:45:11.309880 9766 leader_election.cc:288] T 00000000000000000000000000000000 P 00f35b86d30e401f9d790f52b43b6fc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:45:11.310030 9771 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00f35b86d30e401f9d790f52b43b6fc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:45:11.310717 9771 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00f35b86d30e401f9d790f52b43b6fc6 [term 1 LEADER]: Becoming Leader. State: Replica: 00f35b86d30e401f9d790f52b43b6fc6, State: Running, Role: LEADER
I20260928 08:45:11.311269 9771 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00f35b86d30e401f9d790f52b43b6fc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f35b86d30e401f9d790f52b43b6fc6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45881 } }
I20260928 08:45:11.312129 9766 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00f35b86d30e401f9d790f52b43b6fc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:45:11.315518 9773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00f35b86d30e401f9d790f52b43b6fc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00f35b86d30e401f9d790f52b43b6fc6. Latest consensus state: current_term: 1 leader_uuid: "00f35b86d30e401f9d790f52b43b6fc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f35b86d30e401f9d790f52b43b6fc6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45881 } } }
I20260928 08:45:11.315884 9773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00f35b86d30e401f9d790f52b43b6fc6 [sys.catalog]: This master's current role is: LEADER
I20260928 08:45:11.316072 9772 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00f35b86d30e401f9d790f52b43b6fc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00f35b86d30e401f9d790f52b43b6fc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f35b86d30e401f9d790f52b43b6fc6" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45881 } } }
I20260928 08:45:11.316449 9772 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00f35b86d30e401f9d790f52b43b6fc6 [sys.catalog]: This master's current role is: LEADER
I20260928 08:45:11.317700 9781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:45:11.322732 9781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:45:11.331736 9781 catalog_manager.cc:1347] Generated new cluster ID: 90a8e246214f473a8f512f8a85ac7a6e
I20260928 08:45:11.331875 9781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:45:11.350713 9781 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:45:11.352077 9781 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:45:11.362336 9781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00f35b86d30e401f9d790f52b43b6fc6: Generated new TSK 0
I20260928 08:45:11.363335 9781 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:45:11.376350 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45881
--builtin_ntp_servers=127.0.97.20:42335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 08:45:11.727823 9790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:45:11.728153 9790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:45:11.728366 9790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:45:11.735514 9790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:45:11.735605 9790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:45:11.735769 9790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:45:11.744755 9790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:45:11.746008 9790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:45:11.747776 9790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:45:11.761781 9795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:45:11.761756 9798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:45:11.761724 9796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:45:11.762665 9790 server_base.cc:1034] running on GCE node
I20260928 08:45:11.763409 9790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:45:11.764437 9790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:45:11.765647 9790 hybrid_clock.cc:648] HybridClock initialized: now 1790585111765605 us; error 33 us; skew 500 ppm
I20260928 08:45:11.765981 9790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:45:11.768306 9790 webserver.cc:460] Webserver started at http://127.0.97.1:41459/ using document root <none> and password file <none>
I20260928 08:45:11.769004 9790 fs_manager.cc:359] Metadata directory not provided
I20260928 08:45:11.769133 9790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:45:11.769467 9790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:45:11.772346 9790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-0/data/instance:
uuid: "bf90f7ae684f4e96bd842d7187184294"
format_stamp: "Formatted at 2026-09-28 08:45:11 on dist-test-slave-n6xs"
server_key: "d4f7d40e6e9dd6c8cb2aada96764236a"
server_key_iv: "3120bcaec342f65dec8fca476d177c19"
server_key_version: "encryptionkey@0"
I20260928 08:45:11.773128 9790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-0/wal/instance:
uuid: "bf90f7ae684f4e96bd842d7187184294"
format_stamp: "Formatted at 2026-09-28 08:45:11 on dist-test-slave-n6xs"
server_key: "d4f7d40e6e9dd6c8cb2aada96764236a"
server_key_iv: "3120bcaec342f65dec8fca476d177c19"
server_key_version: "encryptionkey@0"
I20260928 08:45:11.778177 9790 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:45:11.782019 9804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:45:11.783421 9790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:45:11.783622 9790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-0/wal
uuid: "bf90f7ae684f4e96bd842d7187184294"
format_stamp: "Formatted at 2026-09-28 08:45:11 on dist-test-slave-n6xs"
server_key: "d4f7d40e6e9dd6c8cb2aada96764236a"
server_key_iv: "3120bcaec342f65dec8fca476d177c19"
server_key_version: "encryptionkey@0"
I20260928 08:45:11.783883 9790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:45:11.801723 9790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:45:11.802557 9790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:45:11.802847 9790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:45:11.803838 9790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:45:11.805279 9790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:45:11.805398 9790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:45:11.805529 9790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:45:11.805635 9790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:45:11.838953 9790 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37393
I20260928 08:45:11.839003 9916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37393 every 8 connection(s)
I20260928 08:45:11.840386 9790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:45:11.844836 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 9790
I20260928 08:45:11.845031 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:45:11.845680 388 external_mini_cluster.cc:1428] Setting key feddfe2444b7fce2e10087834d4e0940
I20260928 08:45:11.848466 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45881
--builtin_ntp_servers=127.0.97.20:42335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 08:45:11.853351 9919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:11.843079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41401 (local address 127.0.97.62:45881)
0928 08:45:11.843512 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:11.843518 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:11.844891 (+ 1373us) server_negotiation.cc:407] Connection header received
0928 08:45:11.846901 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:11.846906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:11.846993 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:11.847119 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:11.848366 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:11.849196 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:11.850672 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:11.851070 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:11.851197 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:11.852493 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:45:11.852499 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:45:11.852504 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:45:11.852532 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:45:11.852568 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:45:11.852579 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:45:11.852672 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 08:45:11.852916 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 08:45:11.853007 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":208,"threads_started":1}
I20260928 08:45:11.854708 9920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:11.843718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45881 (local address 127.0.97.1:41401)
0928 08:45:11.844721 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:11.844763 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:11.846643 (+ 1880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:11.847382 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:11.847394 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:11.847508 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:11.848208 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:11.848225 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:11.849336 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:11.849343 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:11.850525 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:11.850549 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:11.851280 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:11.851290 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:11.851518 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:11.852193 (+ 675us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:45:11.852196 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:11.852233 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:45:11.852300 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:45:11.852669 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:45:11.852680 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:45:11.852690 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 08:45:11.852895 (+ 205us) client_negotiation.cc:240] Negotiation successful
0928 08:45:11.853289 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":310,"threads_started":1}
I20260928 08:45:11.856515 9917 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45881
I20260928 08:45:11.856869 9917 heartbeater.cc:464] Registering TS with master...
I20260928 08:45:11.857613 9917 heartbeater.cc:511] Master 127.0.97.62:45881 requested a full tablet report, sending...
I20260928 08:45:11.859907 9731 ts_manager.cc:194] Registered new tserver with Master: bf90f7ae684f4e96bd842d7187184294 (127.0.97.1:37393)
I20260928 08:45:11.862185 9731 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:41401
W20260928 08:45:12.171281 9921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:45:12.171522 9921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:45:12.171597 9921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:45:12.178978 9921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:45:12.179080 9921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:45:12.179190 9921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:45:12.188159 9921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:45:12.189443 9921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:45:12.191110 9921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:45:12.204092 9926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:45:12.204134 9927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:45:12.204092 9929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:45:12.209388 9921 server_base.cc:1034] running on GCE node
I20260928 08:45:12.209955 9921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:45:12.210888 9921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:45:12.212075 9921 hybrid_clock.cc:648] HybridClock initialized: now 1790585112212027 us; error 46 us; skew 500 ppm
I20260928 08:45:12.212370 9921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:45:12.214576 9921 webserver.cc:460] Webserver started at http://127.0.97.2:40383/ using document root <none> and password file <none>
I20260928 08:45:12.215366 9921 fs_manager.cc:359] Metadata directory not provided
I20260928 08:45:12.215452 9921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:45:12.215754 9921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:45:12.218627 9921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-1/data/instance:
uuid: "180172d645214f1497d442d0dde88828"
format_stamp: "Formatted at 2026-09-28 08:45:12 on dist-test-slave-n6xs"
server_key: "69ccee8a934fa161cb3c5f6b11266cfc"
server_key_iv: "1a0769dee70d9f8fdddbac9295d40755"
server_key_version: "encryptionkey@0"
I20260928 08:45:12.219373 9921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-1/wal/instance:
uuid: "180172d645214f1497d442d0dde88828"
format_stamp: "Formatted at 2026-09-28 08:45:12 on dist-test-slave-n6xs"
server_key: "69ccee8a934fa161cb3c5f6b11266cfc"
server_key_iv: "1a0769dee70d9f8fdddbac9295d40755"
server_key_version: "encryptionkey@0"
I20260928 08:45:12.224561 9921 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 08:45:12.227994 9935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:45:12.229374 9921 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 08:45:12.229522 9921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-1/wal
uuid: "180172d645214f1497d442d0dde88828"
format_stamp: "Formatted at 2026-09-28 08:45:12 on dist-test-slave-n6xs"
server_key: "69ccee8a934fa161cb3c5f6b11266cfc"
server_key_iv: "1a0769dee70d9f8fdddbac9295d40755"
server_key_version: "encryptionkey@0"
I20260928 08:45:12.229727 9921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:45:12.258049 9921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:45:12.258869 9921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:45:12.259124 9921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:45:12.260097 9921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:45:12.261579 9921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:45:12.261669 9921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:45:12.261740 9921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:45:12.261795 9921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:45:12.296520 9921 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:33329
I20260928 08:45:12.296649 10047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:33329 every 8 connection(s)
I20260928 08:45:12.297829 9921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:45:12.302829 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 9921
I20260928 08:45:12.303017 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:45:12.303432 388 external_mini_cluster.cc:1428] Setting key 43e6c4a0b9658b4be11675413b0c46d6
I20260928 08:45:12.306222 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45881
--builtin_ntp_servers=127.0.97.20:42335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 08:45:12.311344 9919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:12.300449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33097 (local address 127.0.97.62:45881)
0928 08:45:12.300638 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:12.300643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:12.302176 (+ 1533us) server_negotiation.cc:407] Connection header received
0928 08:45:12.304221 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:12.304227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:12.304332 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:12.304488 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:12.305834 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:12.306683 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:12.308413 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:12.308824 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:12.308981 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:12.310409 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:45:12.310414 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:45:12.310418 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:45:12.310436 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:45:12.310467 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:45:12.310480 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:45:12.310590 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 08:45:12.310884 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 08:45:12.310982 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 08:45:12.312496 10050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:12.301044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45881 (local address 127.0.97.2:33097)
0928 08:45:12.301996 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:12.302045 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:12.303933 (+ 1888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:12.304887 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:12.304899 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:12.305038 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:12.305656 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:12.305675 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:12.306847 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:12.306860 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:12.308260 (+ 1400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:12.308273 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:12.309071 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:12.309079 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:12.309339 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:12.310065 (+ 726us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:45:12.310067 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:12.310106 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:45:12.310184 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:45:12.310589 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:45:12.310601 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:45:12.310612 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 08:45:12.310846 (+ 234us) client_negotiation.cc:240] Negotiation successful
0928 08:45:12.311245 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":302,"threads_started":1}
I20260928 08:45:12.313890 10048 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45881
I20260928 08:45:12.314235 10048 heartbeater.cc:464] Registering TS with master...
I20260928 08:45:12.315021 10048 heartbeater.cc:511] Master 127.0.97.62:45881 requested a full tablet report, sending...
I20260928 08:45:12.316555 9731 ts_manager.cc:194] Registered new tserver with Master: 180172d645214f1497d442d0dde88828 (127.0.97.2:33329)
I20260928 08:45:12.317433 9731 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:33097
W20260928 08:45:12.624171 10051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:45:12.624480 10051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:45:12.624621 10051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:45:12.631732 10051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:45:12.631884 10051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:45:12.632054 10051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:45:12.641182 10051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45881
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:45:12.642472 10051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:45:12.644308 10051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:45:12.657228 10056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:45:12.657240 10059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:45:12.657330 10057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:45:12.661322 10051 server_base.cc:1034] running on GCE node
I20260928 08:45:12.661949 10051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:45:12.662947 10051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:45:12.664168 10051 hybrid_clock.cc:648] HybridClock initialized: now 1790585112664139 us; error 73 us; skew 500 ppm
I20260928 08:45:12.664491 10051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:45:12.666734 10051 webserver.cc:460] Webserver started at http://127.0.97.3:39315/ using document root <none> and password file <none>
I20260928 08:45:12.667405 10051 fs_manager.cc:359] Metadata directory not provided
I20260928 08:45:12.667534 10051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:45:12.667918 10051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:45:12.670699 10051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-2/data/instance:
uuid: "dbf88cdffcc94385a8167ddcba22caa2"
format_stamp: "Formatted at 2026-09-28 08:45:12 on dist-test-slave-n6xs"
server_key: "ba721365b8220bd736d84347a0314f95"
server_key_iv: "8c547a7043ea7eac48bd1a193dcf5079"
server_key_version: "encryptionkey@0"
I20260928 08:45:12.671464 10051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-2/wal/instance:
uuid: "dbf88cdffcc94385a8167ddcba22caa2"
format_stamp: "Formatted at 2026-09-28 08:45:12 on dist-test-slave-n6xs"
server_key: "ba721365b8220bd736d84347a0314f95"
server_key_iv: "8c547a7043ea7eac48bd1a193dcf5079"
server_key_version: "encryptionkey@0"
I20260928 08:45:12.676522 10051 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260928 08:45:12.679975 10065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:45:12.681236 10051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 08:45:12.681411 10051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-2/wal
uuid: "dbf88cdffcc94385a8167ddcba22caa2"
format_stamp: "Formatted at 2026-09-28 08:45:12 on dist-test-slave-n6xs"
server_key: "ba721365b8220bd736d84347a0314f95"
server_key_iv: "8c547a7043ea7eac48bd1a193dcf5079"
server_key_version: "encryptionkey@0"
I20260928 08:45:12.681627 10051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:45:12.703131 10051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:45:12.703997 10051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:45:12.704265 10051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:45:12.705190 10051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:45:12.706596 10051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:45:12.706714 10051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:45:12.706851 10051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:45:12.706938 10051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:45:12.740266 10051 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46563
I20260928 08:45:12.740356 10177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46563 every 8 connection(s)
I20260928 08:45:12.741549 10051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:45:12.748049 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 10051
I20260928 08:45:12.748231 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790585022442112-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:45:12.748664 388 external_mini_cluster.cc:1428] Setting key 9058394f920821fd1cf2696d8a1b65bf
I20260928 08:45:12.755088 9919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:12.744241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57115 (local address 127.0.97.62:45881)
0928 08:45:12.744408 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:12.744412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:12.745785 (+ 1373us) server_negotiation.cc:407] Connection header received
0928 08:45:12.747570 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:12.747576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:12.747662 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:12.747786 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:12.748946 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:12.749947 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:12.752371 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:12.752793 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:12.752939 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:12.754282 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:45:12.754287 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:45:12.754292 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:45:12.754318 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:45:12.754346 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:45:12.754357 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:45:12.754446 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 08:45:12.754676 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 08:45:12.754785 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 08:45:12.756279 10180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:12.744760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45881 (local address 127.0.97.3:57115)
0928 08:45:12.745629 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:12.745669 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:12.747307 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:12.748031 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:12.748039 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:12.748130 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:12.748782 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:12.748806 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:12.750112 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:12.750122 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:12.752221 (+ 2099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:12.752233 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:12.753022 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:12.753029 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:12.753270 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:12.753957 (+ 687us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:45:12.753959 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:12.753998 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:45:12.754092 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:45:12.754445 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:45:12.754456 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:45:12.754466 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 08:45:12.754676 (+ 210us) client_negotiation.cc:240] Negotiation successful
0928 08:45:12.755110 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":274,"threads_started":1}
I20260928 08:45:12.757722 10178 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45881
I20260928 08:45:12.758020 10178 heartbeater.cc:464] Registering TS with master...
I20260928 08:45:12.758750 10178 heartbeater.cc:511] Master 127.0.97.62:45881 requested a full tablet report, sending...
I20260928 08:45:12.760239 9731 ts_manager.cc:194] Registered new tserver with Master: dbf88cdffcc94385a8167ddcba22caa2 (127.0.97.3:46563)
I20260928 08:45:12.761013 9731 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:57115
I20260928 08:45:12.764274 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 08:45:12.770495 9919 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:45:12.768729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49886 (local address 127.0.97.62:45881)
0928 08:45:12.768904 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:12.768908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:12.769404 (+ 496us) server_negotiation.cc:407] Connection header received
0928 08:45:12.769529 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:12.769533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:12.769614 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:12.769711 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:12.770232 (+ 521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49886: BlockingRecv error: recv got EOF from 127.0.0.1:49886 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 08:45:12.771405 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 9790
I20260928 08:45:12.779121 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:12.772362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51406 (local address 127.0.97.1:37393)
0928 08:45:12.772862 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:12.772869 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:12.772896 (+ 27us) server_negotiation.cc:407] Connection header received
0928 08:45:12.772952 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:12.772958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:12.773221 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:12.773352 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:12.773673 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:12.774804 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:12.776095 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:12.776679 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:12.776787 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:12.777741 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:45:12.777749 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:45:12.777762 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:45:12.778346 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:45:12.778389 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:45:12.778408 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:45:12.778498 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 08:45:12.778619 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 08:45:12.778775 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":254,"threads_started":1}
I20260928 08:45:12.780429 9912 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:45:12.861435 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 9790
I20260928 08:45:12.877676 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 9921
I20260928 08:45:12.885066 10189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:12.878553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38892 (local address 127.0.97.2:33329)
0928 08:45:12.879095 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:12.879103 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:12.879129 (+ 26us) server_negotiation.cc:407] Connection header received
0928 08:45:12.879192 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:12.879198 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:12.879416 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:12.879547 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:12.879935 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:12.880940 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:12.882092 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:12.882634 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:12.882736 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:12.883617 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:45:12.883625 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:45:12.883638 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:45:12.884317 (+ 679us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:45:12.884360 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:45:12.884378 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:45:12.884464 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 08:45:12.884620 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 08:45:12.884732 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":287,"threads_started":1}
I20260928 08:45:12.886317 10043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:45:12.965763 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 9921
I20260928 08:45:12.981705 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 10051
I20260928 08:45:12.990010 10191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:12.982706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58716 (local address 127.0.97.3:46563)
0928 08:45:12.983361 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:12.983368 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:12.983397 (+ 29us) server_negotiation.cc:407] Connection header received
0928 08:45:12.983455 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:12.983461 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:12.983684 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:12.983874 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:12.984349 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:12.985449 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:12.986747 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:12.987308 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:12.987436 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:12.988640 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:45:12.988647 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:45:12.988662 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:45:12.989290 (+ 628us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:45:12.989336 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:45:12.989354 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:45:12.989448 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 08:45:12.989598 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 08:45:12.989704 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":351,"threads_started":1}
I20260928 08:45:12.991261 10173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:45:13.073963 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 10051
I20260928 08:45:13.091208 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 9700
I20260928 08:45:13.092638 9761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:45:13.196218 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 9700
2026-09-28T08:45:13Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2436 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T08:45:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:45:13Z Disabled control of system clock
I20260928 08:45:13.264407 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35647
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:35647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 08:45:13.586046 10197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:45:13.586325 10197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:45:13.586407 10197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:45:13.593472 10197 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:45:13.593571 10197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:45:13.593607 10197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:45:13.593665 10197 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:45:13.593708 10197 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:45:13.602833 10197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:35647
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:35647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:45:13.604067 10197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:45:13.605801 10197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:45:13.617260 10202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:45:13.617266 10203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:45:13.617307 10205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:45:13.618162 10197 server_base.cc:1034] running on GCE node
I20260928 08:45:13.619011 10197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:45:13.620431 10197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:45:13.621681 10197 hybrid_clock.cc:648] HybridClock initialized: now 1790585113621615 us; error 63 us; skew 500 ppm
I20260928 08:45:13.622072 10197 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:45:13.624490 10197 webserver.cc:460] Webserver started at http://127.0.97.62:42737/ using document root <none> and password file <none>
I20260928 08:45:13.625180 10197 fs_manager.cc:359] Metadata directory not provided
I20260928 08:45:13.625311 10197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:45:13.625624 10197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:45:13.628530 10197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "2df9f828771a47558a7fb318a380e9ad"
format_stamp: "Formatted at 2026-09-28 08:45:13 on dist-test-slave-n6xs"
server_key: "d404f9b67fa382a6a4c3069fc4237a06"
server_key_iv: "5f6a3a16498072cc8bc7ed085a36cde8"
server_key_version: "encryptionkey@0"
I20260928 08:45:13.629312 10197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "2df9f828771a47558a7fb318a380e9ad"
format_stamp: "Formatted at 2026-09-28 08:45:13 on dist-test-slave-n6xs"
server_key: "d404f9b67fa382a6a4c3069fc4237a06"
server_key_iv: "5f6a3a16498072cc8bc7ed085a36cde8"
server_key_version: "encryptionkey@0"
I20260928 08:45:13.634333 10197 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 08:45:13.637900 10211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:45:13.639218 10197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 08:45:13.639394 10197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "2df9f828771a47558a7fb318a380e9ad"
format_stamp: "Formatted at 2026-09-28 08:45:13 on dist-test-slave-n6xs"
server_key: "d404f9b67fa382a6a4c3069fc4237a06"
server_key_iv: "5f6a3a16498072cc8bc7ed085a36cde8"
server_key_version: "encryptionkey@0"
I20260928 08:45:13.639616 10197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:45:13.645673 10197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:45:13.646431 10197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:45:13.646739 10197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:45:13.665047 10197 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:35647
I20260928 08:45:13.665084 10262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:35647 every 8 connection(s)
I20260928 08:45:13.666404 10197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:45:13.670150 10263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:45:13.673389 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 10197
I20260928 08:45:13.673592 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:45:13.674378 388 external_mini_cluster.cc:1428] Setting key fe2ed39c5589a88c8ee92cb5ee09502c
I20260928 08:45:13.682910 10266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:13.676557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49570 (local address 127.0.97.62:35647)
0928 08:45:13.677502 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:13.677514 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:13.677554 (+ 40us) server_negotiation.cc:407] Connection header received
0928 08:45:13.679196 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:13.679218 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:13.679663 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:13.680183 (+ 520us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:13.681306 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:45:13.681312 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:45:13.681324 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:45:13.681336 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:45:13.681391 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:45:13.681405 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:45:13.681632 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 08:45:13.681777 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 08:45:13.682016 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":330,"threads_started":1}
I20260928 08:45:13.682950 10263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2df9f828771a47558a7fb318a380e9ad: Bootstrap starting.
I20260928 08:45:13.686039 10263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2df9f828771a47558a7fb318a380e9ad: Neither blocks nor log segments found. Creating new log.
I20260928 08:45:13.687037 10263 log.cc:824] T 00000000000000000000000000000000 P 2df9f828771a47558a7fb318a380e9ad: Log is configured to *not* fsync() on all Append() calls
I20260928 08:45:13.689813 10263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2df9f828771a47558a7fb318a380e9ad: No bootstrap required, opened a new log
I20260928 08:45:13.697201 10263 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2df9f828771a47558a7fb318a380e9ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2df9f828771a47558a7fb318a380e9ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35647 } }
I20260928 08:45:13.697636 10263 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2df9f828771a47558a7fb318a380e9ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:45:13.697762 10263 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2df9f828771a47558a7fb318a380e9ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2df9f828771a47558a7fb318a380e9ad, State: Initialized, Role: FOLLOWER
I20260928 08:45:13.698331 10263 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2df9f828771a47558a7fb318a380e9ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2df9f828771a47558a7fb318a380e9ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35647 } }
I20260928 08:45:13.698570 10263 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2df9f828771a47558a7fb318a380e9ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:45:13.698697 10263 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2df9f828771a47558a7fb318a380e9ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:45:13.698868 10263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2df9f828771a47558a7fb318a380e9ad [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:45:13.701053 10263 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2df9f828771a47558a7fb318a380e9ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2df9f828771a47558a7fb318a380e9ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35647 } }
I20260928 08:45:13.701614 10263 leader_election.cc:302] T 00000000000000000000000000000000 P 2df9f828771a47558a7fb318a380e9ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2df9f828771a47558a7fb318a380e9ad; no voters:
I20260928 08:45:13.702356 10263 leader_election.cc:288] T 00000000000000000000000000000000 P 2df9f828771a47558a7fb318a380e9ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:45:13.702531 10268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2df9f828771a47558a7fb318a380e9ad [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:45:13.703334 10268 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2df9f828771a47558a7fb318a380e9ad [term 1 LEADER]: Becoming Leader. State: Replica: 2df9f828771a47558a7fb318a380e9ad, State: Running, Role: LEADER
I20260928 08:45:13.703967 10268 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2df9f828771a47558a7fb318a380e9ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2df9f828771a47558a7fb318a380e9ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35647 } }
I20260928 08:45:13.704624 10263 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2df9f828771a47558a7fb318a380e9ad [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:45:13.707083 10270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2df9f828771a47558a7fb318a380e9ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2df9f828771a47558a7fb318a380e9ad. Latest consensus state: current_term: 1 leader_uuid: "2df9f828771a47558a7fb318a380e9ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2df9f828771a47558a7fb318a380e9ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35647 } } }
I20260928 08:45:13.707542 10270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2df9f828771a47558a7fb318a380e9ad [sys.catalog]: This master's current role is: LEADER
I20260928 08:45:13.708056 10269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2df9f828771a47558a7fb318a380e9ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2df9f828771a47558a7fb318a380e9ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2df9f828771a47558a7fb318a380e9ad" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 35647 } } }
I20260928 08:45:13.708461 10269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2df9f828771a47558a7fb318a380e9ad [sys.catalog]: This master's current role is: LEADER
I20260928 08:45:13.711205 10277 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:45:13.716519 10277 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:45:13.724388 10277 catalog_manager.cc:1347] Generated new cluster ID: 2b619b7059d84655a81dc0ee961fd4d8
I20260928 08:45:13.724501 10277 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:45:13.750330 10277 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:45:13.750804 10277 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:45:13.765687 10277 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2df9f828771a47558a7fb318a380e9ad: Generated new TSK 0
I20260928 08:45:13.766672 10277 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:45:13.779081 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35647
--builtin_ntp_servers=127.0.97.20:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 08:45:14.104537 10287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:45:14.104801 10287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:45:14.104952 10287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:45:14.112444 10287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:45:14.112542 10287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:45:14.112699 10287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:45:14.121783 10287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:45:14.123059 10287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:45:14.124835 10287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:45:14.137923 10292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:45:14.137944 10295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:45:14.138028 10293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:45:14.139191 10287 server_base.cc:1034] running on GCE node
I20260928 08:45:14.139811 10287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:45:14.140774 10287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:45:14.141973 10287 hybrid_clock.cc:648] HybridClock initialized: now 1790585114141903 us; error 74 us; skew 500 ppm
I20260928 08:45:14.142411 10287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:45:14.145179 10287 webserver.cc:460] Webserver started at http://127.0.97.1:40591/ using document root <none> and password file <none>
I20260928 08:45:14.145849 10287 fs_manager.cc:359] Metadata directory not provided
I20260928 08:45:14.145961 10287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:45:14.146296 10287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:45:14.150676 10287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-0/data/instance:
uuid: "1e5c7d535e3542ebb3f81231a0ae494f"
format_stamp: "Formatted at 2026-09-28 08:45:14 on dist-test-slave-n6xs"
server_key: "a1fae820b621b914bc9893fa3a014d06"
server_key_iv: "7f83914242a76488379dcffcba2007e7"
server_key_version: "encryptionkey@0"
I20260928 08:45:14.151485 10287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-0/wal/instance:
uuid: "1e5c7d535e3542ebb3f81231a0ae494f"
format_stamp: "Formatted at 2026-09-28 08:45:14 on dist-test-slave-n6xs"
server_key: "a1fae820b621b914bc9893fa3a014d06"
server_key_iv: "7f83914242a76488379dcffcba2007e7"
server_key_version: "encryptionkey@0"
I20260928 08:45:14.156491 10287 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 08:45:14.159855 10301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:45:14.161144 10287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:45:14.161300 10287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-0/wal
uuid: "1e5c7d535e3542ebb3f81231a0ae494f"
format_stamp: "Formatted at 2026-09-28 08:45:14 on dist-test-slave-n6xs"
server_key: "a1fae820b621b914bc9893fa3a014d06"
server_key_iv: "7f83914242a76488379dcffcba2007e7"
server_key_version: "encryptionkey@0"
I20260928 08:45:14.161514 10287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:45:14.167058 10287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:45:14.167811 10287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:45:14.168131 10287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:45:14.168977 10287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:45:14.170271 10287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:45:14.170379 10287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:45:14.170507 10287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:45:14.170607 10287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:45:14.203584 10287 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:37719
I20260928 08:45:14.203619 10413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:37719 every 8 connection(s)
I20260928 08:45:14.205006 10287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:45:14.213760 10416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:14.207738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59817 (local address 127.0.97.62:35647)
0928 08:45:14.208277 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:14.208283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:14.209708 (+ 1425us) server_negotiation.cc:407] Connection header received
0928 08:45:14.211310 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:14.211314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:14.211364 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:14.211464 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:14.212896 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:45:14.212906 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:45:14.212909 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:45:14.212925 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:45:14.212966 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:45:14.212978 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:45:14.213135 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 08:45:14.213305 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 08:45:14.213410 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":263,"threads_started":1}
I20260928 08:45:14.213912 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 10287
I20260928 08:45:14.214087 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:45:14.214572 10417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:14.208534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35647 (local address 127.0.97.1:59817)
0928 08:45:14.209549 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:14.209585 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:14.211060 (+ 1475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:14.211673 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:14.211682 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:14.211775 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:14.212582 (+ 807us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:45:14.212586 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:14.212635 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:45:14.212733 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:45:14.213121 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:45:14.213129 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:45:14.213134 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 08:45:14.213344 (+ 210us) client_negotiation.cc:240] Negotiation successful
0928 08:45:14.213556 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":403,"threads_started":1}
I20260928 08:45:14.214808 388 external_mini_cluster.cc:1428] Setting key 8bd0c20a9c0b933e96b2b9d0102b672c
I20260928 08:45:14.216159 10414 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35647
I20260928 08:45:14.216446 10414 heartbeater.cc:464] Registering TS with master...
I20260928 08:45:14.217173 10414 heartbeater.cc:511] Master 127.0.97.62:35647 requested a full tablet report, sending...
I20260928 08:45:14.217877 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35647
--builtin_ntp_servers=127.0.97.20:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 08:45:14.219244 10228 ts_manager.cc:194] Registered new tserver with Master: 1e5c7d535e3542ebb3f81231a0ae494f (127.0.97.1:37719)
W20260928 08:45:14.540658 10418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:45:14.540934 10418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:45:14.541028 10418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:45:14.548450 10418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:45:14.548560 10418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:45:14.548672 10418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:45:14.557837 10418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:45:14.559131 10418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:45:14.560842 10418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:45:14.574020 10423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:45:14.574103 10426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:45:14.574054 10424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:45:14.574980 10418 server_base.cc:1034] running on GCE node
I20260928 08:45:14.575582 10418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:45:14.576545 10418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:45:14.577772 10418 hybrid_clock.cc:648] HybridClock initialized: now 1790585114577733 us; error 51 us; skew 500 ppm
I20260928 08:45:14.578100 10418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:45:14.580571 10418 webserver.cc:460] Webserver started at http://127.0.97.2:37075/ using document root <none> and password file <none>
I20260928 08:45:14.581384 10418 fs_manager.cc:359] Metadata directory not provided
I20260928 08:45:14.581511 10418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:45:14.581846 10418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:45:14.584776 10418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-1/data/instance:
uuid: "bbd3a3df4623420f8a89af04befa2d97"
format_stamp: "Formatted at 2026-09-28 08:45:14 on dist-test-slave-n6xs"
server_key: "5e59c3b7d9811a653fb5cba3c45fa3b9"
server_key_iv: "662ef78d255c598a48caceb268c8213d"
server_key_version: "encryptionkey@0"
I20260928 08:45:14.585606 10418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-1/wal/instance:
uuid: "bbd3a3df4623420f8a89af04befa2d97"
format_stamp: "Formatted at 2026-09-28 08:45:14 on dist-test-slave-n6xs"
server_key: "5e59c3b7d9811a653fb5cba3c45fa3b9"
server_key_iv: "662ef78d255c598a48caceb268c8213d"
server_key_version: "encryptionkey@0"
I20260928 08:45:14.590627 10418 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 08:45:14.594065 10432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:45:14.595391 10418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 08:45:14.595588 10418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-1/wal
uuid: "bbd3a3df4623420f8a89af04befa2d97"
format_stamp: "Formatted at 2026-09-28 08:45:14 on dist-test-slave-n6xs"
server_key: "5e59c3b7d9811a653fb5cba3c45fa3b9"
server_key_iv: "662ef78d255c598a48caceb268c8213d"
server_key_version: "encryptionkey@0"
I20260928 08:45:14.595857 10418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:45:14.601393 10418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:45:14.602128 10418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:45:14.602414 10418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:45:14.603310 10418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:45:14.604938 10418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:45:14.605072 10418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:45:14.605214 10418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:45:14.605309 10418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:45:14.639073 10418 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40373
I20260928 08:45:14.639115 10544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40373 every 8 connection(s)
I20260928 08:45:14.640575 10418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:45:14.648726 10416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:14.643107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49029 (local address 127.0.97.62:35647)
0928 08:45:14.643326 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:14.643331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:14.644868 (+ 1537us) server_negotiation.cc:407] Connection header received
0928 08:45:14.646546 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:14.646552 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:14.646638 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:14.646748 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:14.648000 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:45:14.648006 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:45:14.648010 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:45:14.648026 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:45:14.648065 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:45:14.648077 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:45:14.648208 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 08:45:14.648406 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 08:45:14.648472 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 08:45:14.649631 10547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:14.643691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35647 (local address 127.0.97.2:49029)
0928 08:45:14.644731 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:14.644766 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:14.646282 (+ 1516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:14.646973 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:14.646981 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:14.647073 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:14.647692 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:45:14.647694 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:14.647728 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:45:14.647843 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:45:14.648169 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:45:14.648180 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:45:14.648188 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 08:45:14.648484 (+ 296us) client_negotiation.cc:240] Negotiation successful
0928 08:45:14.648721 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":275,"threads_started":1}
I20260928 08:45:14.650101 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 10418
I20260928 08:45:14.650326 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:45:14.650924 388 external_mini_cluster.cc:1428] Setting key 7473e99df3ab304f159fe189ee758993
I20260928 08:45:14.651196 10545 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35647
I20260928 08:45:14.651511 10545 heartbeater.cc:464] Registering TS with master...
I20260928 08:45:14.652141 10545 heartbeater.cc:511] Master 127.0.97.62:35647 requested a full tablet report, sending...
I20260928 08:45:14.653538 10228 ts_manager.cc:194] Registered new tserver with Master: bbd3a3df4623420f8a89af04befa2d97 (127.0.97.2:40373)
I20260928 08:45:14.653816 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35647
--builtin_ntp_servers=127.0.97.20:43161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 08:45:14.988770 10548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:45:14.989063 10548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:45:14.989153 10548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:45:14.996690 10548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:45:14.996791 10548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:45:14.996899 10548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:45:15.006460 10548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:43161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:35647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:45:15.007870 10548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:45:15.009711 10548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:45:15.024610 10553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:45:15.024662 10554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:45:15.025429 10556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:45:15.025782 10548 server_base.cc:1034] running on GCE node
I20260928 08:45:15.026696 10548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:45:15.028172 10548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:45:15.029361 10548 hybrid_clock.cc:648] HybridClock initialized: now 1790585115029290 us; error 56 us; skew 500 ppm
I20260928 08:45:15.029820 10548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:45:15.032107 10548 webserver.cc:460] Webserver started at http://127.0.97.3:41863/ using document root <none> and password file <none>
I20260928 08:45:15.032814 10548 fs_manager.cc:359] Metadata directory not provided
I20260928 08:45:15.032927 10548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:45:15.033262 10548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:45:15.036247 10548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-2/data/instance:
uuid: "da39a90b7e774677b6ff5c08793a7b32"
format_stamp: "Formatted at 2026-09-28 08:45:15 on dist-test-slave-n6xs"
server_key: "c13921c1e7647c10db7afa4c128ae0b7"
server_key_iv: "58b98bef89aa615f106c2b385309688f"
server_key_version: "encryptionkey@0"
I20260928 08:45:15.037009 10548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-2/wal/instance:
uuid: "da39a90b7e774677b6ff5c08793a7b32"
format_stamp: "Formatted at 2026-09-28 08:45:15 on dist-test-slave-n6xs"
server_key: "c13921c1e7647c10db7afa4c128ae0b7"
server_key_iv: "58b98bef89aa615f106c2b385309688f"
server_key_version: "encryptionkey@0"
I20260928 08:45:15.041970 10548 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:45:15.045413 10562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:45:15.046794 10548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:45:15.046963 10548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-2/wal
uuid: "da39a90b7e774677b6ff5c08793a7b32"
format_stamp: "Formatted at 2026-09-28 08:45:15 on dist-test-slave-n6xs"
server_key: "c13921c1e7647c10db7afa4c128ae0b7"
server_key_iv: "58b98bef89aa615f106c2b385309688f"
server_key_version: "encryptionkey@0"
I20260928 08:45:15.047188 10548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:45:15.052788 10548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:45:15.053570 10548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:45:15.053834 10548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:45:15.054708 10548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:45:15.056121 10548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:45:15.056209 10548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:45:15.056356 10548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:45:15.056429 10548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:45:15.089771 10548 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43513
I20260928 08:45:15.089869 10674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43513 every 8 connection(s)
I20260928 08:45:15.091107 10548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:45:15.097069 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 10548
I20260928 08:45:15.097257 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790585022442112-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:45:15.097786 388 external_mini_cluster.cc:1428] Setting key eb130bebcd4e563af150d06638a0ca9d
I20260928 08:45:15.099392 10416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:15.093668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44863 (local address 127.0.97.62:35647)
0928 08:45:15.093877 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:15.093884 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:15.095341 (+ 1457us) server_negotiation.cc:407] Connection header received
0928 08:45:15.097114 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:15.097119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:15.097199 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:15.097331 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:15.098671 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:45:15.098682 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:45:15.098692 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:45:15.098708 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:45:15.098736 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:45:15.098747 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:45:15.098888 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 08:45:15.099102 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 08:45:15.099165 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 08:45:15.100939 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:15.094286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:35647 (local address 127.0.97.3:44863)
0928 08:45:15.095173 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:15.095213 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:15.096807 (+ 1594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:15.097559 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:15.097572 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:15.097690 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:15.098378 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:45:15.098380 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:15.098415 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:45:15.098502 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:45:15.098852 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:45:15.098861 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:45:15.098868 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 08:45:15.099213 (+ 345us) client_negotiation.cc:240] Negotiation successful
0928 08:45:15.099489 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":274,"threads_started":1}
I20260928 08:45:15.102372 10675 heartbeater.cc:347] Connected to a master server at 127.0.97.62:35647
I20260928 08:45:15.102625 10675 heartbeater.cc:464] Registering TS with master...
I20260928 08:45:15.103147 10675 heartbeater.cc:511] Master 127.0.97.62:35647 requested a full tablet report, sending...
I20260928 08:45:15.104460 10228 ts_manager.cc:194] Registered new tserver with Master: da39a90b7e774677b6ff5c08793a7b32 (127.0.97.3:43513)
I20260928 08:45:15.113170 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 08:45:15.119361 10416 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:45:15.117652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49578 (local address 127.0.97.62:35647)
0928 08:45:15.117850 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:15.117854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:15.118379 (+ 525us) server_negotiation.cc:407] Connection header received
0928 08:45:15.118484 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:15.118487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:15.118540 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:15.118626 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:15.119077 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49578: BlockingRecv error: recv got EOF from 127.0.0.1:49578 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 08:45:15.119767 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 10287
I20260928 08:45:15.123208 10684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:15.120816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33494 (local address 127.0.97.1:37719)
0928 08:45:15.121321 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:15.121327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:15.121351 (+ 24us) server_negotiation.cc:407] Connection header received
0928 08:45:15.121405 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:15.121410 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:15.121621 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:15.121744 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:15.122634 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:45:15.122642 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:45:15.122653 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:45:15.122666 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:45:15.122709 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:45:15.122724 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:45:15.122825 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 08:45:15.122898 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 08:45:15.122979 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":251,"threads_started":1}
I20260928 08:45:15.124419 10409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:45:15.207002 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 10287
I20260928 08:45:15.223773 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 10418
I20260928 08:45:15.227152 10686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:15.224746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46004 (local address 127.0.97.2:40373)
0928 08:45:15.225221 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:15.225228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:15.225254 (+ 26us) server_negotiation.cc:407] Connection header received
0928 08:45:15.225317 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:15.225323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:15.225555 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:15.225680 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:15.226581 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:45:15.226586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:45:15.226599 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:45:15.226611 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:45:15.226655 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:45:15.226669 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:45:15.226770 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 08:45:15.226851 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 08:45:15.226932 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":220,"threads_started":1}
I20260928 08:45:15.228511 10540 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:45:15.309748 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 10418
I20260928 08:45:15.325322 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 10548
I20260928 08:45:15.329231 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:15.326284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46094 (local address 127.0.97.3:43513)
0928 08:45:15.326848 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:15.326855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:15.326882 (+ 27us) server_negotiation.cc:407] Connection header received
0928 08:45:15.326941 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:15.326947 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:15.327181 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:15.327353 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:15.328502 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:45:15.328508 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:45:15.328520 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:45:15.328531 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:45:15.328569 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:45:15.328583 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:45:15.328767 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 08:45:15.328845 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 08:45:15.328948 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":272,"threads_started":1}
I20260928 08:45:15.330482 10670 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:45:15.411461 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 10548
I20260928 08:45:15.429406 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 10197
I20260928 08:45:15.430639 10258 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:45:15.531483 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 10197
2026-09-28T08:45:15Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2335 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:45:15 dist-test-slave-n6xs krb5kdc[10693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:45:15 dist-test-slave-n6xs krb5kdc[10693](info): set up 2 sockets
Sep 28 08:45:15 dist-test-slave-n6xs krb5kdc[10693](info): commencing operation
krb5kdc: starting...
W20260928 08:45:16.632719 388 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:45:16 dist-test-slave-n6xs krb5kdc[10693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:45:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:45:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:45:16.856058 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36927
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:36927
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:45:17.176172 10706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:45:17.176448 10706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:45:17.176535 10706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:45:17.183578 10706 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:45:17.183688 10706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:45:17.183735 10706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:45:17.183784 10706 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:45:17.183881 10706 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:45:17.193588 10706 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:36927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:36927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:45:17.194849 10706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:45:17.196170 10706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:45:17.207880 10711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:45:17.207885 10712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:45:17.207850 10714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:45:17.209103 10706 server_base.cc:1034] running on GCE node
I20260928 08:45:17.209897 10706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:45:17.211287 10706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:45:17.212479 10706 hybrid_clock.cc:648] HybridClock initialized: now 1790585117212434 us; error 45 us; skew 500 ppm
I20260928 08:45:17.212819 10706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:45:17 dist-test-slave-n6xs krb5kdc[10693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:45:17.215926 10706 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:45:17.217375 10706 webserver.cc:460] Webserver started at http://127.0.97.62:34083/ using document root <none> and password file <none>
I20260928 08:45:17.217995 10706 fs_manager.cc:359] Metadata directory not provided
I20260928 08:45:17.218092 10706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:45:17.218343 10706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:45:17.221261 10706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "777557133373418fa0e34ecd39c95fd3"
format_stamp: "Formatted at 2026-09-28 08:45:17 on dist-test-slave-n6xs"
server_key: "c3fbaba7ae34eabbc0f292ab873bd6d1"
server_key_iv: "28d170ba2b0a92081de1a81f9fade0fa"
server_key_version: "encryptionkey@0"
I20260928 08:45:17.221928 10706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "777557133373418fa0e34ecd39c95fd3"
format_stamp: "Formatted at 2026-09-28 08:45:17 on dist-test-slave-n6xs"
server_key: "c3fbaba7ae34eabbc0f292ab873bd6d1"
server_key_iv: "28d170ba2b0a92081de1a81f9fade0fa"
server_key_version: "encryptionkey@0"
I20260928 08:45:17.226854 10706 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 08:45:17.230170 10721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:45:17.231501 10706 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 08:45:17.231676 10706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "777557133373418fa0e34ecd39c95fd3"
format_stamp: "Formatted at 2026-09-28 08:45:17 on dist-test-slave-n6xs"
server_key: "c3fbaba7ae34eabbc0f292ab873bd6d1"
server_key_iv: "28d170ba2b0a92081de1a81f9fade0fa"
server_key_version: "encryptionkey@0"
I20260928 08:45:17.231894 10706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:45:17.253149 10706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:45:17.262897 10706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:45:17.263208 10706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:45:17.281454 10706 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:36927
I20260928 08:45:17.281505 10772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:36927 every 8 connection(s)
I20260928 08:45:17.282765 10706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:45:17.286437 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 10706
I20260928 08:45:17.286607 10773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:45:17.286646 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:45:17.287497 388 external_mini_cluster.cc:1428] Setting key e9d1818d841ec091ead8b881ad11fcfb
I20260928 08:45:17.300370 10773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 777557133373418fa0e34ecd39c95fd3: Bootstrap starting.
Sep 28 08:45:17 dist-test-slave-n6xs krb5kdc[10693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:45:17.304117 10773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 777557133373418fa0e34ecd39c95fd3: Neither blocks nor log segments found. Creating new log.
I20260928 08:45:17.305181 10773 log.cc:824] T 00000000000000000000000000000000 P 777557133373418fa0e34ecd39c95fd3: Log is configured to *not* fsync() on all Append() calls
I20260928 08:45:17.308576 10773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 777557133373418fa0e34ecd39c95fd3: No bootstrap required, opened a new log
I20260928 08:45:17.312461 10776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:17.289522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57468 (local address 127.0.97.62:36927)
0928 08:45:17.290312 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:17.290323 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:17.290361 (+ 38us) server_negotiation.cc:407] Connection header received
0928 08:45:17.292622 (+ 2261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:17.292643 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:17.293020 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:17.293353 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:17.295163 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:17.296583 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:17.298432 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:17.299093 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:17.299222 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:17.303942 (+ 4720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:45:17.303950 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:45:17.303966 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:45:17.304014 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:45:17.307877 (+ 3863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:45:17.308574 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:45:17.308582 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:45:17.308591 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:45:17.308785 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:45:17.309166 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:45:17.309171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:45:17.309174 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:45:17.309888 (+ 714us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:45:17.310193 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0928 08:45:17.310584 (+ 391us) server_negotiation.cc:299] Negotiation successful
0928 08:45:17.311041 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":391,"threads_started":1}
I20260928 08:45:17.317876 10773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 777557133373418fa0e34ecd39c95fd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "777557133373418fa0e34ecd39c95fd3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36927 } }
I20260928 08:45:17.318277 10773 raft_consensus.cc:374] T 00000000000000000000000000000000 P 777557133373418fa0e34ecd39c95fd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:45:17.318389 10773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 777557133373418fa0e34ecd39c95fd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 777557133373418fa0e34ecd39c95fd3, State: Initialized, Role: FOLLOWER
I20260928 08:45:17.319037 10773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 777557133373418fa0e34ecd39c95fd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "777557133373418fa0e34ecd39c95fd3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36927 } }
I20260928 08:45:17.319283 10773 raft_consensus.cc:388] T 00000000000000000000000000000000 P 777557133373418fa0e34ecd39c95fd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:45:17.319423 10773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 777557133373418fa0e34ecd39c95fd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:45:17.319607 10773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 777557133373418fa0e34ecd39c95fd3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:45:17.321743 10773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 777557133373418fa0e34ecd39c95fd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "777557133373418fa0e34ecd39c95fd3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36927 } }
I20260928 08:45:17.322234 10773 leader_election.cc:302] T 00000000000000000000000000000000 P 777557133373418fa0e34ecd39c95fd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 777557133373418fa0e34ecd39c95fd3; no voters:
I20260928 08:45:17.322803 10773 leader_election.cc:288] T 00000000000000000000000000000000 P 777557133373418fa0e34ecd39c95fd3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:45:17.323025 10778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 777557133373418fa0e34ecd39c95fd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:45:17.323869 10778 raft_consensus.cc:686] T 00000000000000000000000000000000 P 777557133373418fa0e34ecd39c95fd3 [term 1 LEADER]: Becoming Leader. State: Replica: 777557133373418fa0e34ecd39c95fd3, State: Running, Role: LEADER
I20260928 08:45:17.324381 10778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 777557133373418fa0e34ecd39c95fd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "777557133373418fa0e34ecd39c95fd3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36927 } }
I20260928 08:45:17.324922 10773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 777557133373418fa0e34ecd39c95fd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:45:17.327338 10779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 777557133373418fa0e34ecd39c95fd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "777557133373418fa0e34ecd39c95fd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "777557133373418fa0e34ecd39c95fd3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36927 } } }
I20260928 08:45:17.327847 10779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 777557133373418fa0e34ecd39c95fd3 [sys.catalog]: This master's current role is: LEADER
I20260928 08:45:17.327371 10780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 777557133373418fa0e34ecd39c95fd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 777557133373418fa0e34ecd39c95fd3. Latest consensus state: current_term: 1 leader_uuid: "777557133373418fa0e34ecd39c95fd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "777557133373418fa0e34ecd39c95fd3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 36927 } } }
I20260928 08:45:17.328416 10780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 777557133373418fa0e34ecd39c95fd3 [sys.catalog]: This master's current role is: LEADER
I20260928 08:45:17.330639 10787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:45:17.338646 10787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:45:17.347138 10787 catalog_manager.cc:1347] Generated new cluster ID: e76f6ceb0b6b4cb1b614c9f05deae07f
I20260928 08:45:17.347262 10787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:45:17.360592 10787 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:45:17.362416 10787 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:45:17.370040 10787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 777557133373418fa0e34ecd39c95fd3: Generated new TSK 0
I20260928 08:45:17.371294 10787 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:45:17.451674 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36927
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:45:17.775027 10799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:45:17.775352 10799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:45:17.775529 10799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:45:17.782872 10799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:45:17.782985 10799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:45:17.783098 10799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:45:17.792733 10799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:45:17.793968 10799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:45:17.795204 10799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:45:17.807863 10804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:45:17.808010 10805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:45:17.808145 10807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:45:17.809216 10799 server_base.cc:1034] running on GCE node
I20260928 08:45:17.809808 10799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:45:17.810746 10799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:45:17.811960 10799 hybrid_clock.cc:648] HybridClock initialized: now 1790585117811870 us; error 95 us; skew 500 ppm
I20260928 08:45:17.812319 10799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:45:17 dist-test-slave-n6xs krb5kdc[10693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:45:17.815454 10799 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:45:17.816804 10799 webserver.cc:460] Webserver started at http://127.0.97.1:45361/ using document root <none> and password file <none>
I20260928 08:45:17.817482 10799 fs_manager.cc:359] Metadata directory not provided
I20260928 08:45:17.817615 10799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:45:17.817916 10799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:45:17.820822 10799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-0/data/instance:
uuid: "3b1f258ecb7a42539118863d2a63ae8e"
format_stamp: "Formatted at 2026-09-28 08:45:17 on dist-test-slave-n6xs"
server_key: "3c7b31b4dd155823dc8c331ad21c2587"
server_key_iv: "2b2bc2ec547a898b9902c753b7a420f5"
server_key_version: "encryptionkey@0"
I20260928 08:45:17.821569 10799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-0/wal/instance:
uuid: "3b1f258ecb7a42539118863d2a63ae8e"
format_stamp: "Formatted at 2026-09-28 08:45:17 on dist-test-slave-n6xs"
server_key: "3c7b31b4dd155823dc8c331ad21c2587"
server_key_iv: "2b2bc2ec547a898b9902c753b7a420f5"
server_key_version: "encryptionkey@0"
I20260928 08:45:17.826558 10799 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 08:45:17.830006 10814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:45:17.831277 10799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:45:17.831456 10799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-0/wal
uuid: "3b1f258ecb7a42539118863d2a63ae8e"
format_stamp: "Formatted at 2026-09-28 08:45:17 on dist-test-slave-n6xs"
server_key: "3c7b31b4dd155823dc8c331ad21c2587"
server_key_iv: "2b2bc2ec547a898b9902c753b7a420f5"
server_key_version: "encryptionkey@0"
I20260928 08:45:17.831640 10799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:45:17.864638 10799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:45:17.869091 10799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:45:17.869385 10799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:45:17.870390 10799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:45:17.871693 10799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:45:17.871762 10799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:45:17.871955 10799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:45:17.872020 10799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:45:17.904850 10799 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40109
I20260928 08:45:17.904938 10926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40109 every 8 connection(s)
I20260928 08:45:17.906103 10799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:45:17.913869 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 10799
I20260928 08:45:17.914083 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:45:17.914702 388 external_mini_cluster.cc:1428] Setting key 16511b9ef73f7209f6a61930f8360fad
Sep 28 08:45:17 dist-test-slave-n6xs krb5kdc[10693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:45:17.926700 10929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:17.908750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40235 (local address 127.0.97.62:36927)
0928 08:45:17.909211 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:17.909215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:17.910152 (+ 937us) server_negotiation.cc:407] Connection header received
0928 08:45:17.912037 (+ 1885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:17.912042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:17.912134 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:17.912253 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:17.913747 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:17.914569 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:17.916364 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:17.916866 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:17.916981 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:17.921163 (+ 4182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:45:17.921168 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:45:17.921172 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:45:17.921200 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:45:17.923661 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:45:17.924398 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:45:17.924404 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:45:17.924409 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:45:17.924520 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:45:17.924937 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:45:17.924947 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:45:17.924950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:45:17.925304 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:45:17.925445 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 08:45:17.926114 (+ 669us) server_negotiation.cc:299] Negotiation successful
0928 08:45:17.926332 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":244,"threads_started":1}
I20260928 08:45:17.927680 10930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:17.909325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36927 (local address 127.0.97.1:40235)
0928 08:45:17.909995 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:17.910034 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:17.911704 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:17.912498 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:17.912506 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:17.913003 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:17.913566 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:17.913582 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:17.914730 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:17.914736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:17.916209 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:17.916226 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:17.917127 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:17.917134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:17.917453 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:17.918204 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:45:17.918244 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:17.920919 (+ 2675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:45:17.923796 (+ 2877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:45:17.923831 (+ 35us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:45:17.923849 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:45:17.924262 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:45:17.924626 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:45:17.924631 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:45:17.924635 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:45:17.924815 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:45:17.925443 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:45:17.925451 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:45:17.925761 (+ 310us) client_negotiation.cc:769] Sending connection context
0928 08:45:17.926014 (+ 253us) client_negotiation.cc:240] Negotiation successful
0928 08:45:17.926605 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":257,"threads_started":1}
I20260928 08:45:17.929248 10927 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36927
I20260928 08:45:17.929572 10927 heartbeater.cc:464] Registering TS with master...
I20260928 08:45:17.930327 10927 heartbeater.cc:511] Master 127.0.97.62:36927 requested a full tablet report, sending...
I20260928 08:45:17.932442 10738 ts_manager.cc:194] Registered new tserver with Master: 3b1f258ecb7a42539118863d2a63ae8e (127.0.97.1:40109)
I20260928 08:45:17.933853 10738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40235
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 08:45:17.967028 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36927
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:45:18.289687 10933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:45:18.290052 10933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:45:18.290223 10933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:45:18.297737 10933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:45:18.297886 10933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:45:18.298060 10933 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:45:18.307967 10933 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:45:18.309263 10933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:45:18.310592 10933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:45:18.323840 10938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:45:18.323933 10941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:45:18.323881 10939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:45:18.324903 10933 server_base.cc:1034] running on GCE node
I20260928 08:45:18.325573 10933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:45:18.326638 10933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:45:18.327931 10933 hybrid_clock.cc:648] HybridClock initialized: now 1790585118327848 us; error 47 us; skew 500 ppm
I20260928 08:45:18.328279 10933 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:45:18 dist-test-slave-n6xs krb5kdc[10693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:45:18.331424 10933 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:45:18.332893 10933 webserver.cc:460] Webserver started at http://127.0.97.2:37063/ using document root <none> and password file <none>
I20260928 08:45:18.333597 10933 fs_manager.cc:359] Metadata directory not provided
I20260928 08:45:18.333740 10933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:45:18.334046 10933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:45:18.337224 10933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-1/data/instance:
uuid: "e8d5e6f9c98d4122a61e41146a1bae42"
format_stamp: "Formatted at 2026-09-28 08:45:18 on dist-test-slave-n6xs"
server_key: "b421036323f60a9f3649e487db7d7275"
server_key_iv: "48df6c378cbaf4b4755cd83dca6fd7f3"
server_key_version: "encryptionkey@0"
I20260928 08:45:18.338006 10933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-1/wal/instance:
uuid: "e8d5e6f9c98d4122a61e41146a1bae42"
format_stamp: "Formatted at 2026-09-28 08:45:18 on dist-test-slave-n6xs"
server_key: "b421036323f60a9f3649e487db7d7275"
server_key_iv: "48df6c378cbaf4b4755cd83dca6fd7f3"
server_key_version: "encryptionkey@0"
I20260928 08:45:18.343228 10933 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260928 08:45:18.346755 10948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:45:18.348143 10933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:45:18.348309 10933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-1/wal
uuid: "e8d5e6f9c98d4122a61e41146a1bae42"
format_stamp: "Formatted at 2026-09-28 08:45:18 on dist-test-slave-n6xs"
server_key: "b421036323f60a9f3649e487db7d7275"
server_key_iv: "48df6c378cbaf4b4755cd83dca6fd7f3"
server_key_version: "encryptionkey@0"
I20260928 08:45:18.348515 10933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:45:18.378280 10933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:45:18.382556 10933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:45:18.382849 10933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:45:18.383880 10933 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:45:18.385221 10933 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:45:18.385324 10933 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:45:18.385473 10933 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:45:18.385555 10933 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:45:18.418740 10933 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35921
I20260928 08:45:18.418803 11060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35921 every 8 connection(s)
I20260928 08:45:18.420169 10933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:45:18.429037 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 10933
I20260928 08:45:18.429201 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:45:18.429672 388 external_mini_cluster.cc:1428] Setting key 9e0b294909dc20b51c63ceadf157585f
Sep 28 08:45:18 dist-test-slave-n6xs krb5kdc[10693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:45:18.441169 10929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:18.422659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:33425 (local address 127.0.97.62:36927)
0928 08:45:18.422832 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:18.422836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:18.423989 (+ 1153us) server_negotiation.cc:407] Connection header received
0928 08:45:18.425747 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:18.425752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:18.425820 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:18.425955 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:18.427294 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:18.428057 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:18.429847 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:18.430412 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:18.430528 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:18.434379 (+ 3851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:45:18.434388 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:45:18.434396 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:45:18.434438 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:45:18.438049 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:45:18.438729 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:45:18.438737 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:45:18.438743 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:45:18.438899 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:45:18.439285 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:45:18.439293 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:45:18.439299 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:45:18.439677 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:45:18.439786 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 08:45:18.440742 (+ 956us) server_negotiation.cc:299] Negotiation successful
0928 08:45:18.440932 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 08:45:18.442655 11063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:18.423092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36927 (local address 127.0.97.2:33425)
0928 08:45:18.423773 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:18.423863 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:18.425424 (+ 1561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:18.426129 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:18.426137 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:18.426622 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:18.427135 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:18.427151 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:18.428188 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:18.428192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:18.429708 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:18.429720 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:18.430606 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:18.430632 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:18.430900 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:18.431684 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:45:18.431714 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:18.434150 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:45:18.438204 (+ 4054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:45:18.438210 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:45:18.438223 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:45:18.438611 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:45:18.439005 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:45:18.439008 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:45:18.439011 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:45:18.439178 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:45:18.439824 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:45:18.439834 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:45:18.440324 (+ 490us) client_negotiation.cc:769] Sending connection context
0928 08:45:18.440644 (+ 320us) client_negotiation.cc:240] Negotiation successful
0928 08:45:18.441215 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":299,"threads_started":1}
I20260928 08:45:18.444146 11061 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36927
I20260928 08:45:18.444437 11061 heartbeater.cc:464] Registering TS with master...
I20260928 08:45:18.445122 11061 heartbeater.cc:511] Master 127.0.97.62:36927 requested a full tablet report, sending...
I20260928 08:45:18.446527 10738 ts_manager.cc:194] Registered new tserver with Master: e8d5e6f9c98d4122a61e41146a1bae42 (127.0.97.2:35921)
I20260928 08:45:18.447739 10738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:33425
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:45:18.483170 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36927
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:45:18.804453 11066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:45:18.804739 11066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:45:18.804906 11066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:45:18.812312 11066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:45:18.812420 11066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:45:18.812631 11066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:45:18.822405 11066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:36927
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:45:18.823704 11066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:45:18.825101 11066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:45:18.838975 11071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:45:18.839042 11074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:45:18.839064 11072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:45:18.839982 11066 server_base.cc:1034] running on GCE node
I20260928 08:45:18.840528 11066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:45:18.841410 11066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:45:18.842623 11066 hybrid_clock.cc:648] HybridClock initialized: now 1790585118842558 us; error 57 us; skew 500 ppm
I20260928 08:45:18.842895 11066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:45:18 dist-test-slave-n6xs krb5kdc[10693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:45:18.846130 11066 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:45:18.847671 11066 webserver.cc:460] Webserver started at http://127.0.97.3:46403/ using document root <none> and password file <none>
I20260928 08:45:18.848438 11066 fs_manager.cc:359] Metadata directory not provided
I20260928 08:45:18.848551 11066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:45:18.848812 11066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:45:18.852030 11066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-2/data/instance:
uuid: "52a026e75285430e82d7770766adc24b"
format_stamp: "Formatted at 2026-09-28 08:45:18 on dist-test-slave-n6xs"
server_key: "27142a6a66339850f276e45851ae7227"
server_key_iv: "96b6e35b3c56da1fb2ee0106c30861d0"
server_key_version: "encryptionkey@0"
I20260928 08:45:18.852852 11066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-2/wal/instance:
uuid: "52a026e75285430e82d7770766adc24b"
format_stamp: "Formatted at 2026-09-28 08:45:18 on dist-test-slave-n6xs"
server_key: "27142a6a66339850f276e45851ae7227"
server_key_iv: "96b6e35b3c56da1fb2ee0106c30861d0"
server_key_version: "encryptionkey@0"
I20260928 08:45:18.858211 11066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:45:18.861688 11081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:45:18.863122 11066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:45:18.863252 11066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-2/wal
uuid: "52a026e75285430e82d7770766adc24b"
format_stamp: "Formatted at 2026-09-28 08:45:18 on dist-test-slave-n6xs"
server_key: "27142a6a66339850f276e45851ae7227"
server_key_iv: "96b6e35b3c56da1fb2ee0106c30861d0"
server_key_version: "encryptionkey@0"
I20260928 08:45:18.863405 11066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:45:18.879191 11066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:45:18.883266 11066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:45:18.883507 11066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:45:18.884519 11066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:45:18.885823 11066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:45:18.885895 11066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:45:18.885988 11066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:45:18.886040 11066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:45:18.919845 11066 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:40639
I20260928 08:45:18.920009 11193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:40639 every 8 connection(s)
I20260928 08:45:18.921144 11066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:45:18.925618 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 11066
I20260928 08:45:18.925781 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:45:18.926311 388 external_mini_cluster.cc:1428] Setting key 0d3e00404c19b27ad85cce727b84580d
Sep 28 08:45:18 dist-test-slave-n6xs krb5kdc[10693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:45:18.937620 10927 heartbeater.cc:503] Master 127.0.97.62:36927 was elected leader, sending a full tablet report...
I20260928 08:45:18.942843 10929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:18.923701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:45777 (local address 127.0.97.62:36927)
0928 08:45:18.923935 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:18.923941 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:18.925177 (+ 1236us) server_negotiation.cc:407] Connection header received
0928 08:45:18.927333 (+ 2156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:18.927337 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:18.927402 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:18.927543 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:18.929632 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:18.930470 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:18.932067 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:18.932792 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:18.932948 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:18.936691 (+ 3743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:45:18.936696 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:45:18.936700 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:45:18.936730 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:45:18.939355 (+ 2625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:45:18.940242 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:45:18.940249 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:45:18.940254 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:45:18.940446 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:45:18.940998 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:45:18.941004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:45:18.941007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:45:18.941407 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:45:18.941565 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 08:45:18.942417 (+ 852us) server_negotiation.cc:299] Negotiation successful
0928 08:45:18.942591 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 08:45:18.943858 11196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:18.924211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:36927 (local address 127.0.97.3:45777)
0928 08:45:18.925026 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:18.925066 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:18.927068 (+ 2002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:18.927847 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:18.927862 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:18.928652 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:45:18.929463 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:18.929490 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:18.930605 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:18.930609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:18.931857 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:18.931874 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:18.933026 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:18.933033 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:18.933336 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:18.934044 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:45:18.934065 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:45:18.936502 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:45:18.939495 (+ 2993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:45:18.939501 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:45:18.939515 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:45:18.940038 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:45:18.940578 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:45:18.940582 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:45:18.940585 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:45:18.940862 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:45:18.941568 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:45:18.941577 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:45:18.941999 (+ 422us) client_negotiation.cc:769] Sending connection context
0928 08:45:18.942281 (+ 282us) client_negotiation.cc:240] Negotiation successful
0928 08:45:18.942657 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":279,"threads_started":1}
I20260928 08:45:18.945291 11194 heartbeater.cc:347] Connected to a master server at 127.0.97.62:36927
I20260928 08:45:18.945608 11194 heartbeater.cc:464] Registering TS with master...
I20260928 08:45:18.946311 11194 heartbeater.cc:511] Master 127.0.97.62:36927 requested a full tablet report, sending...
I20260928 08:45:18.947721 10738 ts_manager.cc:194] Registered new tserver with Master: 52a026e75285430e82d7770766adc24b (127.0.97.3:40639)
I20260928 08:45:18.948578 10738 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:45777
I20260928 08:45:18.954675 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:45:18.971253 10929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:18.959606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57484 (local address 127.0.97.62:36927)
0928 08:45:18.959783 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:18.959787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:18.960223 (+ 436us) server_negotiation.cc:407] Connection header received
0928 08:45:18.960355 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:18.960359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:18.960417 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:18.960500 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:18.961822 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:18.962516 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:18.963755 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:18.964163 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:18.964251 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:18.966403 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:45:18.966408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:45:18.966412 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:45:18.966442 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:45:18.969108 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:45:18.969726 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:45:18.969729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:45:18.969731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:45:18.969839 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:45:18.970151 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:45:18.970155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:45:18.970157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:45:18.970433 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:45:18.970607 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 08:45:18.970863 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 08:45:18.971021 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 08:45:18.975239 10738 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57484:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:45:18.981988 10738 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:45:19.006762 11206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:18.993209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40639 (local address 127.0.0.1:44688)
0928 08:45:18.994013 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:18.994056 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:18.994205 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:18.996664 (+ 2459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:18.996674 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:18.996715 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:45:18.997202 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:18.997214 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:19.000001 (+ 2787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:19.000010 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:19.002876 (+ 2866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:19.002896 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:19.005701 (+ 2805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:19.005713 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:19.006009 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:19.006092 (+ 83us) client_negotiation.cc:769] Sending connection context
0928 08:45:19.006261 (+ 169us) client_negotiation.cc:240] Negotiation successful
0928 08:45:19.006468 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":438,"threads_started":1}
I20260928 08:45:19.007941 11208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:18.993368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44688 (local address 127.0.97.3:40639)
0928 08:45:18.995876 (+ 2508us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:18.995888 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:18.995941 (+ 53us) server_negotiation.cc:407] Connection header received
0928 08:45:18.996067 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:18.996078 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:18.996384 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:18.997409 (+ 1025us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:45:18.997687 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:18.999742 (+ 2055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:19.003166 (+ 3424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:19.005120 (+ 1954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:19.005358 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:19.005615 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0928 08:45:19.007405 (+ 1790us) server_negotiation.cc:299] Negotiation successful
0928 08:45:19.007575 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2279,"thread_start_us":2113,"threads_started":1}
I20260928 08:45:19.011901 11129 tablet_service.cc:1467] Processing CreateTablet for tablet f70cf2e7d47e4652b41d58c3368d40ab (DEFAULT_TABLE table=test-table [id=35c12909f392456a9ee8f2cd2b2a3464]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:45:19.013290 11207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:18.993364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35921 (local address 127.0.0.1:59124)
0928 08:45:18.995048 (+ 1684us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:18.995066 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:18.995185 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:19.006921 (+ 11736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:19.006931 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:19.006959 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:45:19.007362 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:19.007372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:19.009038 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:19.009046 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:19.010767 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:19.010779 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:19.012535 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:19.012541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:19.012694 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:19.012726 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 08:45:19.012828 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 08:45:19.012955 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1577,"thread_start_us":1428,"threads_started":1}
I20260928 08:45:19.013372 11209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:18.994644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59124 (local address 127.0.97.2:35921)
0928 08:45:19.006148 (+ 11504us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:19.006157 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:19.006192 (+ 35us) server_negotiation.cc:407] Connection header received
0928 08:45:19.006306 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:19.006313 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:19.006674 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:19.007562 (+ 888us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:45:19.007791 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:19.008892 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:19.010942 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:19.012341 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:19.012477 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:19.012656 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 08:45:19.012890 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 08:45:19.013040 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11369,"thread_start_us":11271,"threads_started":1}
I20260928 08:45:19.013695 11129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f70cf2e7d47e4652b41d58c3368d40ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:45:19.014814 11210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:18.995139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40109 (local address 127.0.0.1:47642)
0928 08:45:19.007065 (+ 11926us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:19.007089 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:19.007216 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:19.007732 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:19.007737 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:19.007762 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:45:19.008951 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:19.008966 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:19.010871 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:19.010879 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:19.012201 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:19.012212 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:19.014049 (+ 1837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:19.014055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:19.014220 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:19.014264 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 08:45:19.014348 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 08:45:19.014487 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11758,"thread_start_us":10242,"threads_started":1}
I20260928 08:45:19.015589 11211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:18.995518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47642 (local address 127.0.97.1:40109)
0928 08:45:19.006642 (+ 11124us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:19.006650 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:19.007175 (+ 525us) server_negotiation.cc:407] Connection header received
0928 08:45:19.007372 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:19.007378 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:19.007548 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:19.007745 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:45:19.009184 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:19.010730 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:19.012351 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:19.013854 (+ 1503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:19.014004 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:19.014220 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 08:45:19.015051 (+ 831us) server_negotiation.cc:299] Negotiation successful
0928 08:45:19.015211 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10969,"thread_start_us":10738,"threads_started":1}
I20260928 08:45:19.017179 10996 tablet_service.cc:1467] Processing CreateTablet for tablet f70cf2e7d47e4652b41d58c3368d40ab (DEFAULT_TABLE table=test-table [id=35c12909f392456a9ee8f2cd2b2a3464]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:45:19.018328 10996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f70cf2e7d47e4652b41d58c3368d40ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:45:19.019692 10862 tablet_service.cc:1467] Processing CreateTablet for tablet f70cf2e7d47e4652b41d58c3368d40ab (DEFAULT_TABLE table=test-table [id=35c12909f392456a9ee8f2cd2b2a3464]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:45:19.021016 10862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f70cf2e7d47e4652b41d58c3368d40ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:45:19.027894 11212 tablet_bootstrap.cc:492] T f70cf2e7d47e4652b41d58c3368d40ab P 52a026e75285430e82d7770766adc24b: Bootstrap starting.
I20260928 08:45:19.032498 11212 tablet_bootstrap.cc:654] T f70cf2e7d47e4652b41d58c3368d40ab P 52a026e75285430e82d7770766adc24b: Neither blocks nor log segments found. Creating new log.
I20260928 08:45:19.034193 11212 log.cc:824] T f70cf2e7d47e4652b41d58c3368d40ab P 52a026e75285430e82d7770766adc24b: Log is configured to *not* fsync() on all Append() calls
I20260928 08:45:19.034443 11214 tablet_bootstrap.cc:492] T f70cf2e7d47e4652b41d58c3368d40ab P 3b1f258ecb7a42539118863d2a63ae8e: Bootstrap starting.
I20260928 08:45:19.035981 11213 tablet_bootstrap.cc:492] T f70cf2e7d47e4652b41d58c3368d40ab P e8d5e6f9c98d4122a61e41146a1bae42: Bootstrap starting.
I20260928 08:45:19.038007 11214 tablet_bootstrap.cc:654] T f70cf2e7d47e4652b41d58c3368d40ab P 3b1f258ecb7a42539118863d2a63ae8e: Neither blocks nor log segments found. Creating new log.
I20260928 08:45:19.039007 11212 tablet_bootstrap.cc:492] T f70cf2e7d47e4652b41d58c3368d40ab P 52a026e75285430e82d7770766adc24b: No bootstrap required, opened a new log
I20260928 08:45:19.039175 11214 log.cc:824] T f70cf2e7d47e4652b41d58c3368d40ab P 3b1f258ecb7a42539118863d2a63ae8e: Log is configured to *not* fsync() on all Append() calls
I20260928 08:45:19.039289 11212 ts_tablet_manager.cc:1397] T f70cf2e7d47e4652b41d58c3368d40ab P 52a026e75285430e82d7770766adc24b: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260928 08:45:19.039440 11213 tablet_bootstrap.cc:654] T f70cf2e7d47e4652b41d58c3368d40ab P e8d5e6f9c98d4122a61e41146a1bae42: Neither blocks nor log segments found. Creating new log.
I20260928 08:45:19.040468 11213 log.cc:824] T f70cf2e7d47e4652b41d58c3368d40ab P e8d5e6f9c98d4122a61e41146a1bae42: Log is configured to *not* fsync() on all Append() calls
I20260928 08:45:19.042099 11214 tablet_bootstrap.cc:492] T f70cf2e7d47e4652b41d58c3368d40ab P 3b1f258ecb7a42539118863d2a63ae8e: No bootstrap required, opened a new log
I20260928 08:45:19.042343 11214 ts_tablet_manager.cc:1397] T f70cf2e7d47e4652b41d58c3368d40ab P 3b1f258ecb7a42539118863d2a63ae8e: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 08:45:19.043309 11213 tablet_bootstrap.cc:492] T f70cf2e7d47e4652b41d58c3368d40ab P e8d5e6f9c98d4122a61e41146a1bae42: No bootstrap required, opened a new log
I20260928 08:45:19.043545 11213 ts_tablet_manager.cc:1397] T f70cf2e7d47e4652b41d58c3368d40ab P e8d5e6f9c98d4122a61e41146a1bae42: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260928 08:45:19.047468 11212 raft_consensus.cc:348] T f70cf2e7d47e4652b41d58c3368d40ab P 52a026e75285430e82d7770766adc24b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a026e75285430e82d7770766adc24b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40639 } } peers { permanent_uuid: "e8d5e6f9c98d4122a61e41146a1bae42" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35921 } } peers { permanent_uuid: "3b1f258ecb7a42539118863d2a63ae8e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40109 } }
I20260928 08:45:19.047861 11212 raft_consensus.cc:374] T f70cf2e7d47e4652b41d58c3368d40ab P 52a026e75285430e82d7770766adc24b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:45:19.047961 11212 raft_consensus.cc:729] T f70cf2e7d47e4652b41d58c3368d40ab P 52a026e75285430e82d7770766adc24b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52a026e75285430e82d7770766adc24b, State: Initialized, Role: FOLLOWER
I20260928 08:45:19.048602 11212 consensus_queue.cc:260] T f70cf2e7d47e4652b41d58c3368d40ab P 52a026e75285430e82d7770766adc24b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a026e75285430e82d7770766adc24b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40639 } } peers { permanent_uuid: "e8d5e6f9c98d4122a61e41146a1bae42" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35921 } } peers { permanent_uuid: "3b1f258ecb7a42539118863d2a63ae8e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40109 } }
I20260928 08:45:19.049706 11194 heartbeater.cc:503] Master 127.0.97.62:36927 was elected leader, sending a full tablet report...
I20260928 08:45:19.050376 11212 ts_tablet_manager.cc:1428] T f70cf2e7d47e4652b41d58c3368d40ab P 52a026e75285430e82d7770766adc24b: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260928 08:45:19.053467 11214 raft_consensus.cc:348] T f70cf2e7d47e4652b41d58c3368d40ab P 3b1f258ecb7a42539118863d2a63ae8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a026e75285430e82d7770766adc24b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40639 } } peers { permanent_uuid: "e8d5e6f9c98d4122a61e41146a1bae42" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35921 } } peers { permanent_uuid: "3b1f258ecb7a42539118863d2a63ae8e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40109 } }
I20260928 08:45:19.053927 11214 raft_consensus.cc:374] T f70cf2e7d47e4652b41d58c3368d40ab P 3b1f258ecb7a42539118863d2a63ae8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:45:19.054032 11214 raft_consensus.cc:729] T f70cf2e7d47e4652b41d58c3368d40ab P 3b1f258ecb7a42539118863d2a63ae8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b1f258ecb7a42539118863d2a63ae8e, State: Initialized, Role: FOLLOWER
I20260928 08:45:19.053932 11213 raft_consensus.cc:348] T f70cf2e7d47e4652b41d58c3368d40ab P e8d5e6f9c98d4122a61e41146a1bae42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a026e75285430e82d7770766adc24b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40639 } } peers { permanent_uuid: "e8d5e6f9c98d4122a61e41146a1bae42" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35921 } } peers { permanent_uuid: "3b1f258ecb7a42539118863d2a63ae8e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40109 } }
I20260928 08:45:19.054345 11213 raft_consensus.cc:374] T f70cf2e7d47e4652b41d58c3368d40ab P e8d5e6f9c98d4122a61e41146a1bae42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:45:19.054431 11213 raft_consensus.cc:729] T f70cf2e7d47e4652b41d58c3368d40ab P e8d5e6f9c98d4122a61e41146a1bae42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8d5e6f9c98d4122a61e41146a1bae42, State: Initialized, Role: FOLLOWER
I20260928 08:45:19.054685 11214 consensus_queue.cc:260] T f70cf2e7d47e4652b41d58c3368d40ab P 3b1f258ecb7a42539118863d2a63ae8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a026e75285430e82d7770766adc24b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40639 } } peers { permanent_uuid: "e8d5e6f9c98d4122a61e41146a1bae42" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35921 } } peers { permanent_uuid: "3b1f258ecb7a42539118863d2a63ae8e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40109 } }
I20260928 08:45:19.054978 11213 consensus_queue.cc:260] T f70cf2e7d47e4652b41d58c3368d40ab P e8d5e6f9c98d4122a61e41146a1bae42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a026e75285430e82d7770766adc24b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40639 } } peers { permanent_uuid: "e8d5e6f9c98d4122a61e41146a1bae42" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35921 } } peers { permanent_uuid: "3b1f258ecb7a42539118863d2a63ae8e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40109 } }
I20260928 08:45:19.056957 11214 ts_tablet_manager.cc:1428] T f70cf2e7d47e4652b41d58c3368d40ab P 3b1f258ecb7a42539118863d2a63ae8e: Time spent starting tablet: real 0.014s user 0.015s sys 0.001s
I20260928 08:45:19.057996 11061 heartbeater.cc:503] Master 127.0.97.62:36927 was elected leader, sending a full tablet report...
I20260928 08:45:19.058388 11213 ts_tablet_manager.cc:1428] T f70cf2e7d47e4652b41d58c3368d40ab P e8d5e6f9c98d4122a61e41146a1bae42: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260928 08:45:19.151418 11220 raft_consensus.cc:482] T f70cf2e7d47e4652b41d58c3368d40ab P e8d5e6f9c98d4122a61e41146a1bae42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:45:19.151695 11220 raft_consensus.cc:504] T f70cf2e7d47e4652b41d58c3368d40ab P e8d5e6f9c98d4122a61e41146a1bae42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a026e75285430e82d7770766adc24b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40639 } } peers { permanent_uuid: "e8d5e6f9c98d4122a61e41146a1bae42" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35921 } } peers { permanent_uuid: "3b1f258ecb7a42539118863d2a63ae8e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40109 } }
I20260928 08:45:19.153538 11220 leader_election.cc:288] T f70cf2e7d47e4652b41d58c3368d40ab P e8d5e6f9c98d4122a61e41146a1bae42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52a026e75285430e82d7770766adc24b (127.0.97.3:40639), 3b1f258ecb7a42539118863d2a63ae8e (127.0.97.1:40109)
W20260928 08:45:19.158463 10928 tablet.cc:2369] T f70cf2e7d47e4652b41d58c3368d40ab P 3b1f258ecb7a42539118863d2a63ae8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:45:19.161388 11222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:19.153885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:40639 (local address 127.0.97.2:34649)
0928 08:45:19.154868 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:19.154883 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:19.155047 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:19.155456 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:19.155459 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:19.155491 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:45:19.155987 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:19.155997 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:19.157134 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:19.157137 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:19.158756 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:19.158768 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:19.160534 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:19.160541 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:19.160716 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:19.160757 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 08:45:19.160851 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 08:45:19.160981 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":886,"mutex_wait_us":17,"thread_start_us":466,"threads_started":1}
I20260928 08:45:19.161473 11211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:19.154086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:60341 (local address 127.0.97.1:40109)
0928 08:45:19.154337 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:19.154342 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:19.154839 (+ 497us) server_negotiation.cc:407] Connection header received
0928 08:45:19.155109 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:19.155118 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:19.155202 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:19.155269 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:45:19.156083 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:19.156932 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:19.159110 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:19.160349 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:19.160521 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:19.160617 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 08:45:19.161109 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 08:45:19.161237 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 08:45:19.161531 11208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:19.153841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34649 (local address 127.0.97.3:40639)
0928 08:45:19.154047 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:19.154055 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:19.154990 (+ 935us) server_negotiation.cc:407] Connection header received
0928 08:45:19.155210 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:19.155214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:19.155322 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:19.155426 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:45:19.156156 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:19.156996 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:19.158901 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:19.160120 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:19.160246 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:19.160350 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 08:45:19.160967 (+ 617us) server_negotiation.cc:299] Negotiation successful
0928 08:45:19.161153 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 08:45:19.162221 11221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:19.153798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40109 (local address 127.0.97.2:60341)
0928 08:45:19.154738 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0928 08:45:19.154754 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:45:19.154916 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:45:19.155335 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:45:19.155339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:45:19.155384 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:45:19.155939 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:19.155951 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:19.157051 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:19.157058 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:19.158957 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:45:19.158971 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:45:19.160753 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:45:19.160758 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:45:19.160941 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:19.160971 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 08:45:19.161691 (+ 720us) client_negotiation.cc:240] Negotiation successful
0928 08:45:19.161838 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"mutex_wait_us":56,"thread_start_us":441,"threads_started":1}
I20260928 08:45:19.163038 11149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f70cf2e7d47e4652b41d58c3368d40ab" candidate_uuid: "e8d5e6f9c98d4122a61e41146a1bae42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a026e75285430e82d7770766adc24b" is_pre_election: true
I20260928 08:45:19.163609 11149 raft_consensus.cc:2454] T f70cf2e7d47e4652b41d58c3368d40ab P 52a026e75285430e82d7770766adc24b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8d5e6f9c98d4122a61e41146a1bae42 in term 0.
I20260928 08:45:19.163537 10882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f70cf2e7d47e4652b41d58c3368d40ab" candidate_uuid: "e8d5e6f9c98d4122a61e41146a1bae42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b1f258ecb7a42539118863d2a63ae8e" is_pre_election: true
I20260928 08:45:19.164187 10882 raft_consensus.cc:2454] T f70cf2e7d47e4652b41d58c3368d40ab P 3b1f258ecb7a42539118863d2a63ae8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8d5e6f9c98d4122a61e41146a1bae42 in term 0.
I20260928 08:45:19.164412 10952 leader_election.cc:302] T f70cf2e7d47e4652b41d58c3368d40ab P e8d5e6f9c98d4122a61e41146a1bae42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52a026e75285430e82d7770766adc24b, e8d5e6f9c98d4122a61e41146a1bae42; no voters:
I20260928 08:45:19.164846 11220 raft_consensus.cc:2781] T f70cf2e7d47e4652b41d58c3368d40ab P e8d5e6f9c98d4122a61e41146a1bae42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:45:19.165055 11220 raft_consensus.cc:482] T f70cf2e7d47e4652b41d58c3368d40ab P e8d5e6f9c98d4122a61e41146a1bae42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:45:19.165158 11220 raft_consensus.cc:3037] T f70cf2e7d47e4652b41d58c3368d40ab P e8d5e6f9c98d4122a61e41146a1bae42 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:45:19.167402 11220 raft_consensus.cc:504] T f70cf2e7d47e4652b41d58c3368d40ab P e8d5e6f9c98d4122a61e41146a1bae42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a026e75285430e82d7770766adc24b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40639 } } peers { permanent_uuid: "e8d5e6f9c98d4122a61e41146a1bae42" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35921 } } peers { permanent_uuid: "3b1f258ecb7a42539118863d2a63ae8e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40109 } }
I20260928 08:45:19.168048 11220 leader_election.cc:288] T f70cf2e7d47e4652b41d58c3368d40ab P e8d5e6f9c98d4122a61e41146a1bae42 [CANDIDATE]: Term 1 election: Requested vote from peers 52a026e75285430e82d7770766adc24b (127.0.97.3:40639), 3b1f258ecb7a42539118863d2a63ae8e (127.0.97.1:40109)
I20260928 08:45:19.168371 11149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f70cf2e7d47e4652b41d58c3368d40ab" candidate_uuid: "e8d5e6f9c98d4122a61e41146a1bae42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52a026e75285430e82d7770766adc24b"
I20260928 08:45:19.168520 10882 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f70cf2e7d47e4652b41d58c3368d40ab" candidate_uuid: "e8d5e6f9c98d4122a61e41146a1bae42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b1f258ecb7a42539118863d2a63ae8e"
I20260928 08:45:19.168587 11149 raft_consensus.cc:3037] T f70cf2e7d47e4652b41d58c3368d40ab P 52a026e75285430e82d7770766adc24b [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:45:19.168718 10882 raft_consensus.cc:3037] T f70cf2e7d47e4652b41d58c3368d40ab P 3b1f258ecb7a42539118863d2a63ae8e [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:45:19.171770 10882 raft_consensus.cc:2454] T f70cf2e7d47e4652b41d58c3368d40ab P 3b1f258ecb7a42539118863d2a63ae8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8d5e6f9c98d4122a61e41146a1bae42 in term 1.
I20260928 08:45:19.171763 11149 raft_consensus.cc:2454] T f70cf2e7d47e4652b41d58c3368d40ab P 52a026e75285430e82d7770766adc24b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8d5e6f9c98d4122a61e41146a1bae42 in term 1.
W20260928 08:45:19.172087 11062 tablet.cc:2369] T f70cf2e7d47e4652b41d58c3368d40ab P e8d5e6f9c98d4122a61e41146a1bae42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:45:19.172154 10952 leader_election.cc:302] T f70cf2e7d47e4652b41d58c3368d40ab P e8d5e6f9c98d4122a61e41146a1bae42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52a026e75285430e82d7770766adc24b, e8d5e6f9c98d4122a61e41146a1bae42; no voters:
I20260928 08:45:19.172497 11220 raft_consensus.cc:2781] T f70cf2e7d47e4652b41d58c3368d40ab P e8d5e6f9c98d4122a61e41146a1bae42 [term 1 FOLLOWER]: Leader election won for term 1
W20260928 08:45:19.172710 11195 tablet.cc:2369] T f70cf2e7d47e4652b41d58c3368d40ab P 52a026e75285430e82d7770766adc24b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:45:19.173028 11220 raft_consensus.cc:686] T f70cf2e7d47e4652b41d58c3368d40ab P e8d5e6f9c98d4122a61e41146a1bae42 [term 1 LEADER]: Becoming Leader. State: Replica: e8d5e6f9c98d4122a61e41146a1bae42, State: Running, Role: LEADER
I20260928 08:45:19.173540 11220 consensus_queue.cc:237] T f70cf2e7d47e4652b41d58c3368d40ab P e8d5e6f9c98d4122a61e41146a1bae42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a026e75285430e82d7770766adc24b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40639 } } peers { permanent_uuid: "e8d5e6f9c98d4122a61e41146a1bae42" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35921 } } peers { permanent_uuid: "3b1f258ecb7a42539118863d2a63ae8e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40109 } }
I20260928 08:45:19.177834 10737 catalog_manager.cc:5495] T f70cf2e7d47e4652b41d58c3368d40ab P e8d5e6f9c98d4122a61e41146a1bae42 reported cstate change: term changed from 0 to 1, leader changed from <none> to e8d5e6f9c98d4122a61e41146a1bae42 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "e8d5e6f9c98d4122a61e41146a1bae42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52a026e75285430e82d7770766adc24b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8d5e6f9c98d4122a61e41146a1bae42" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35921 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b1f258ecb7a42539118863d2a63ae8e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40109 } health_report { overall_health: UNKNOWN } } }
I20260928 08:45:19.195883 11209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:19.190752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59126 (local address 127.0.97.2:35921)
0928 08:45:19.190964 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:19.190969 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:19.190985 (+ 16us) server_negotiation.cc:407] Connection header received
0928 08:45:19.191061 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:19.191065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:19.191135 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:19.191257 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:45:19.191641 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:19.192437 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:19.193740 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:19.194227 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:19.194335 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:19.194800 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:45:19.195335 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:45:19.195457 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 08:45:19.195522 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 08:45:19.195585 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 08:45:19.203668 10882 raft_consensus.cc:1261] T f70cf2e7d47e4652b41d58c3368d40ab P 3b1f258ecb7a42539118863d2a63ae8e [term 1 FOLLOWER]: Refusing update from remote peer e8d5e6f9c98d4122a61e41146a1bae42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:45:19.203791 11149 raft_consensus.cc:1261] T f70cf2e7d47e4652b41d58c3368d40ab P 52a026e75285430e82d7770766adc24b [term 1 FOLLOWER]: Refusing update from remote peer e8d5e6f9c98d4122a61e41146a1bae42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:45:19.204798 11223 consensus_queue.cc:1035] T f70cf2e7d47e4652b41d58c3368d40ab P e8d5e6f9c98d4122a61e41146a1bae42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b1f258ecb7a42539118863d2a63ae8e" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:45:19.205245 11220 consensus_queue.cc:1035] T f70cf2e7d47e4652b41d58c3368d40ab P e8d5e6f9c98d4122a61e41146a1bae42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52a026e75285430e82d7770766adc24b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 40639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:45:19.216230 11226 mvcc.cc:205] Tried to move back new op lower bound from 7334236648249098240 to 7334236648138481664. Current Snapshot: MvccSnapshot[applied={T|T < 7334236648249098240}]
I20260928 08:45:19.216787 11228 mvcc.cc:205] Tried to move back new op lower bound from 7334236648249098240 to 7334236648138481664. Current Snapshot: MvccSnapshot[applied={T|T < 7334236648249098240}]
I20260928 08:45:19.229470 10737 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:45:19.229784 10737 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57484:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:45:19.233175 10737 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 777557133373418fa0e34ecd39c95fd3: Sending DeleteTablet for 3 replicas of tablet f70cf2e7d47e4652b41d58c3368d40ab
I20260928 08:45:19.234231 11129 tablet_service.cc:1514] Processing DeleteTablet for tablet f70cf2e7d47e4652b41d58c3368d40ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:45:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:44688
I20260928 08:45:19.234306 10996 tablet_service.cc:1514] Processing DeleteTablet for tablet f70cf2e7d47e4652b41d58c3368d40ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:45:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:59124
I20260928 08:45:19.234504 10862 tablet_service.cc:1514] Processing DeleteTablet for tablet f70cf2e7d47e4652b41d58c3368d40ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:45:19 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47642
I20260928 08:45:19.241583 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 10799
I20260928 08:45:19.247179 11237 tablet_replica.cc:330] T f70cf2e7d47e4652b41d58c3368d40ab P e8d5e6f9c98d4122a61e41146a1bae42: stopping tablet replica
I20260928 08:45:19.247960 11237 raft_consensus.cc:2229] T f70cf2e7d47e4652b41d58c3368d40ab P e8d5e6f9c98d4122a61e41146a1bae42 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:45:19.248910 11237 raft_consensus.cc:2258] T f70cf2e7d47e4652b41d58c3368d40ab P e8d5e6f9c98d4122a61e41146a1bae42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:45:19.249878 11238 tablet_replica.cc:330] T f70cf2e7d47e4652b41d58c3368d40ab P 3b1f258ecb7a42539118863d2a63ae8e: stopping tablet replica
I20260928 08:45:19.250833 11236 tablet_replica.cc:330] T f70cf2e7d47e4652b41d58c3368d40ab P 52a026e75285430e82d7770766adc24b: stopping tablet replica
I20260928 08:45:19.251245 11236 raft_consensus.cc:2229] T f70cf2e7d47e4652b41d58c3368d40ab P 52a026e75285430e82d7770766adc24b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:45:19.251745 11236 raft_consensus.cc:2258] T f70cf2e7d47e4652b41d58c3368d40ab P 52a026e75285430e82d7770766adc24b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:45:19.252629 11237 ts_tablet_manager.cc:1905] T f70cf2e7d47e4652b41d58c3368d40ab P e8d5e6f9c98d4122a61e41146a1bae42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:45:19.253799 11236 ts_tablet_manager.cc:1905] T f70cf2e7d47e4652b41d58c3368d40ab P 52a026e75285430e82d7770766adc24b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:45:19.252187 11238 raft_consensus.cc:2229] T f70cf2e7d47e4652b41d58c3368d40ab P 3b1f258ecb7a42539118863d2a63ae8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:45:19.259267 11238 raft_consensus.cc:2258] T f70cf2e7d47e4652b41d58c3368d40ab P 3b1f258ecb7a42539118863d2a63ae8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:45:19.260064 11237 ts_tablet_manager.cc:1918] T f70cf2e7d47e4652b41d58c3368d40ab P e8d5e6f9c98d4122a61e41146a1bae42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:45:19.260286 11237 log.cc:1196] T f70cf2e7d47e4652b41d58c3368d40ab P e8d5e6f9c98d4122a61e41146a1bae42: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-1/wal/wals/f70cf2e7d47e4652b41d58c3368d40ab
I20260928 08:45:19.260459 11236 ts_tablet_manager.cc:1918] T f70cf2e7d47e4652b41d58c3368d40ab P 52a026e75285430e82d7770766adc24b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:45:19.260622 11236 log.cc:1196] T f70cf2e7d47e4652b41d58c3368d40ab P 52a026e75285430e82d7770766adc24b: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-2/wal/wals/f70cf2e7d47e4652b41d58c3368d40ab
I20260928 08:45:19.261025 11237 ts_tablet_manager.cc:1939] T f70cf2e7d47e4652b41d58c3368d40ab P e8d5e6f9c98d4122a61e41146a1bae42: Deleting consensus metadata
I20260928 08:45:19.261107 11236 ts_tablet_manager.cc:1939] T f70cf2e7d47e4652b41d58c3368d40ab P 52a026e75285430e82d7770766adc24b: Deleting consensus metadata
I20260928 08:45:19.263016 11238 ts_tablet_manager.cc:1905] T f70cf2e7d47e4652b41d58c3368d40ab P 3b1f258ecb7a42539118863d2a63ae8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:45:19.263389 10725 catalog_manager.cc:4841] TS 52a026e75285430e82d7770766adc24b (127.0.97.3:40639): tablet f70cf2e7d47e4652b41d58c3368d40ab (table test-table [id=35c12909f392456a9ee8f2cd2b2a3464]) successfully deleted
Sep 28 08:45:19 dist-test-slave-n6xs krb5kdc[10693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:45:19.265954 10722 catalog_manager.cc:4841] TS e8d5e6f9c98d4122a61e41146a1bae42 (127.0.97.2:35921): tablet f70cf2e7d47e4652b41d58c3368d40ab (table test-table [id=35c12909f392456a9ee8f2cd2b2a3464]) successfully deleted
I20260928 08:45:19.269965 11238 ts_tablet_manager.cc:1918] T f70cf2e7d47e4652b41d58c3368d40ab P 3b1f258ecb7a42539118863d2a63ae8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:45:19.270120 11238 log.cc:1196] T f70cf2e7d47e4652b41d58c3368d40ab P 3b1f258ecb7a42539118863d2a63ae8e: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790585022442112-388-0/minicluster-data/ts-0/wal/wals/f70cf2e7d47e4652b41d58c3368d40ab
I20260928 08:45:19.270632 11238 ts_tablet_manager.cc:1939] T f70cf2e7d47e4652b41d58c3368d40ab P 3b1f258ecb7a42539118863d2a63ae8e: Deleting consensus metadata
I20260928 08:45:19.272398 10724 catalog_manager.cc:4841] TS 3b1f258ecb7a42539118863d2a63ae8e (127.0.97.1:40109): tablet f70cf2e7d47e4652b41d58c3368d40ab (table test-table [id=35c12909f392456a9ee8f2cd2b2a3464]) successfully deleted
I20260928 08:45:19.272737 11211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:19.244963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47644 (local address 127.0.97.1:40109)
0928 08:45:19.245141 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:19.245147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:19.250189 (+ 5042us) server_negotiation.cc:407] Connection header received
0928 08:45:19.250398 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:19.250403 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:19.250491 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:19.250592 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:19.252417 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:19.256865 (+ 4448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:19.259481 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:19.260200 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:19.260299 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:19.266315 (+ 6016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:45:19.266330 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:45:19.266348 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:45:19.266414 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:45:19.269712 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:45:19.270498 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:45:19.270506 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:45:19.270509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:45:19.270649 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:45:19.271087 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:45:19.271091 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:45:19.271094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:45:19.271460 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:45:19.271775 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0928 08:45:19.272101 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 08:45:19.272309 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 08:45:19.273366 10922 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:45:19.376299 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 10799
I20260928 08:45:19.399616 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 10933
Sep 28 08:45:19 dist-test-slave-n6xs krb5kdc[10693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 08:45:19.417275 11209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:19.400497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59142 (local address 127.0.97.2:35921)
0928 08:45:19.400673 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:19.400681 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:19.400706 (+ 25us) server_negotiation.cc:407] Connection header received
0928 08:45:19.400971 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:19.400975 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:19.401048 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:19.401166 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:19.402349 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:19.403169 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:19.404651 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:19.405101 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:19.405251 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:19.409163 (+ 3912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:45:19.409181 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:45:19.409198 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:45:19.409241 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:45:19.413121 (+ 3880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:45:19.414133 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:45:19.414142 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:45:19.414146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:45:19.414376 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:45:19.414933 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:45:19.414938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:45:19.414941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:45:19.415551 (+ 610us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:45:19.415932 (+ 381us) server_negotiation.cc:1035] Waiting for connection context
0928 08:45:19.416395 (+ 463us) server_negotiation.cc:299] Negotiation successful
0928 08:45:19.416708 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 08:45:19.419035 11056 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:45:19.531898 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 10933
I20260928 08:45:19.550871 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 11066
Sep 28 08:45:19 dist-test-slave-n6xs krb5kdc[10693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 08:45:19.565238 11208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:45:19.551665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44698 (local address 127.0.97.3:40639)
0928 08:45:19.551960 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0928 08:45:19.551965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:45:19.551983 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:45:19.552125 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:45:19.552128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:45:19.552191 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:45:19.552274 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:45:19.553538 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:19.554249 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:19.555524 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:45:19.556101 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:45:19.556199 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:45:19.560151 (+ 3952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:45:19.560166 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:45:19.560182 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:45:19.560218 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:45:19.562652 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:45:19.563314 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:45:19.563320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:45:19.563322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:45:19.563456 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:45:19.563748 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:45:19.563751 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:45:19.563753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:45:19.564243 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:45:19.564463 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 08:45:19.564718 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 08:45:19.564862 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 08:45:19.565995 11189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:45:19.654346 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 11066
I20260928 08:45:19.672492 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 10706
I20260928 08:45:19.673596 10768 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:45:19.779595 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 10706
2026-09-28T08:45:19Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4248 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:45:19 dist-test-slave-n6xs krb5kdc[11246](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:45:19 dist-test-slave-n6xs krb5kdc[11246](info): set up 2 sockets
Sep 28 08:45:19 dist-test-slave-n6xs krb5kdc[11246](info): commencing operation
krb5kdc: starting...
W20260928 08:45:20.872761 388 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:45:21 dist-test-slave-n6xs krb5kdc[11246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:45:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:45:21Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260928 08:45:21.195212 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskUfDo5M/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/postgres -l logfile start
2026-09-28 08:45:23.746 UTC [11276] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 08:45:23.746 UTC [11276] LOG: listening on IPv4 address "127.0.97.20", port 38841
2026-09-28 08:45:23.750 UTC [11276] LOG: listening on Unix socket "/tmp/.s.PGSQL.38841"
2026-09-28 08:45:23.771 UTC [11279] LOG: database system was shut down at 2026-09-28 08:45:22 UTC
2026-09-28 08:45:23.775 UTC [11276] LOG: database system is ready to accept connections
I20260928 08:45:24.744628 388 mini_postgres.cc:96] Postgres bound to 38841
2026-09-28 08:45:24.762 UTC [11287] FATAL: database "slave" does not exist
127.0.97.20:38841 - accepting connections
I20260928 08:45:24.763322 388 mini_ranger.cc:161] Starting Ranger...
I20260928 08:45:24.790611 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 08:45:25.301916 388 mini_ranger.cc:87] Created ranger Postgres database
I20260928 08:45:25.302081 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ranger-admin
2026-09-28 08:45:25,607 [I] DB FLAVOR :POSTGRES
2026-09-28 08:45:25,607 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 08:45:25,608 [I] Checking connection..
2026-09-28 08:45:25,608 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 08:45:25,870 [I] Checking connection passed.
2026-09-28 08:45:25,870 [I] --------- Verifying version history table ---------
2026-09-28 08:45:25,870 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 08:45:26,114 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 08:45:26,115 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 08:45:26,389 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 08:45:26,389 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 08:45:26,390 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 08:45:26.641 UTC [11351] WARNING: there is no transaction in progress
2026-09-28 08:45:26,648 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 08:45:26,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 08:45:26,894 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 08:45:26,895 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 08:45:26,895 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 08:45:27,163 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 08:45:27,424 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-n6xs.c.gcp-upstream.internal','N') ;"
2026-09-28 08:45:27,690 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 08:45:27,958 [I] Table x_portal_user does not exist in database ranger
2026-09-28 08:45:27,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 08:45:28,212 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 08:45:28,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 08:45:28,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 08:45:28,718 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 08:45:28,719 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 08:45:29.726 UTC [11459] WARNING: there is no transaction in progress
2026-09-28 08:45:29.742 UTC [11459] WARNING: there is no transaction in progress
2026-09-28 08:45:29.758 UTC [11459] WARNING: there is no transaction in progress
2026-09-28 08:45:29.774 UTC [11459] WARNING: there is no transaction in progress
2026-09-28 08:45:29.790 UTC [11459] WARNING: there is no transaction in progress
2026-09-28 08:45:29.808 UTC [11459] WARNING: there is no transaction in progress
2026-09-28 08:45:29.865 UTC [11459] WARNING: there is no transaction in progress
2026-09-28 08:45:29.872 UTC [11459] WARNING: there is no transaction in progress
2026-09-28 08:45:29.886 UTC [11459] WARNING: there is no transaction in progress
2026-09-28 08:45:29.896 UTC [11459] WARNING: there is no transaction in progress
2026-09-28 08:45:29.906 UTC [11459] WARNING: there is no transaction in progress
2026-09-28 08:45:29.920 UTC [11459] WARNING: there is no transaction in progress
2026-09-28 08:45:29.930 UTC [11459] WARNING: there is no transaction in progress
2026-09-28 08:45:29.936 UTC [11459] WARNING: there is no transaction in progress
2026-09-28 08:45:29.944 UTC [11459] WARNING: there is no transaction in progress
2026-09-28 08:45:29.950 UTC [11459] WARNING: there is no transaction in progress
2026-09-28 08:45:30,593 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 08:45:30,594 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 08:45:30,837 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 08:45:30,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 08:45:31,101 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 08:45:31,101 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 08:45:31,348 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 08:45:31,348 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 08:45:31,609 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 08:45:31,852 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-n6xs.c.gcp-upstream.internal';"
2026-09-28 08:45:32,101 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 08:45:32,101 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 08:45:32,338 [I] DB_PATCHES have already been applied
I20260928 08:45:32.343871 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskUfDo5M/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 08:45:32.343973 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260928 08:45:32.358223 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:55441
Sep 28, 2026 8:45:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 8:45:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 8:45:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 8:45:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 8:45:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 8:45:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 8:45:33 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 08:45:33.493487 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.730s user 0.000s sys 0.039s
Sep 28, 2026 8:45:53 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 8:45:53 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 8:45:53 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 8:45:53 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 8:45:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 8:45:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 8:45:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 8:45:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 8:45:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 8:45:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 8:45:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 8:45:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 8:45:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 8:45:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 8:45:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 8:45:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 8:45:53 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 8:45:54 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 08:45:54.643512 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 08:45:54 dist-test-slave-n6xs krb5kdc[11246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585154, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 08:45:54.823124 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 08:45:54.843894 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 08:45:55.742352 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 08:45:55.742547 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ranger-kms
I20260928 08:45:55.742705 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:38841
2026-09-28 08:45:56,156 [I] DB FLAVOR :POSTGRES
2026-09-28 08:45:56,157 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 08:45:56,157 [I] Checking connection
2026-09-28 08:45:56,157 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 08:45:56,417 [I] connection success
2026-09-28 08:45:56,417 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 08:45:56,418 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 08:45:56,418 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 08:45:56,647 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 08:45:56,648 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 08:45:56,648 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 08:45:56,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:38841/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 08:45:56,906 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 08:45:57.325248 388 mini_ranger_kms.cc:310] Created kms service
I20260928 08:45:57.529520 388 mini_ranger_kms.cc:326] Created kudu user
I20260928 08:45:57.576855 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 08:45:57 dist-test-slave-n6xs krb5kdc[11246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585154, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 08:45:58.022239 388 mini_ranger_kms.cc:384] Added ranger policy
I20260928 08:45:58.022558 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 08:45:58.022632 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260928 08:45:58.039887 388 mini_ranger_kms.cc:276] Ranger KMS PID: 11639
I20260928 08:45:58.040067 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:42953
Sep 28, 2026 8:45:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 8:45:59 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 8:45:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 8:45:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 8:45:59 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 8:45:59 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 08:46:00.359965 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.537s user 0.108s sys 0.326s
I20260928 08:46:00.360220 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 08:46:00.360433 388 mini_ranger_kms.cc:401] 127.0.97.20:42953/kms/v1/keys
Sep 28 08:46:00 dist-test-slave-n6xs krb5kdc[11246](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790585160, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 08:46:00 dist-test-slave-n6xs krb5kdc[11246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585160, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 28 08:46:01 dist-test-slave-n6xs krb5kdc[11246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585160, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 28 08:46:03 dist-test-slave-n6xs krb5kdc[11246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:46:03.580034 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41113
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41113
--ranger_config_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:42953/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:46:03.920403 11690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:46:03.920761 11690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:46:03.920871 11690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:46:03.928021 11690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:46:03.928126 11690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:46:03.928174 11690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:46:03.928233 11690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:46:03.928287 11690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:46:03.938788 11690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:42953/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41113
--ranger_config_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:46:03.940045 11690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:46:03.941340 11690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:46:03.953594 11695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:46:03.953707 11696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:46:03.953673 11698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:46:03.955309 11690 server_base.cc:1034] running on GCE node
I20260928 08:46:03.956127 11690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:46:03.957485 11690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:46:03.958703 11690 hybrid_clock.cc:648] HybridClock initialized: now 1790585163958664 us; error 26 us; skew 500 ppm
I20260928 08:46:03.959004 11690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:46:03 dist-test-slave-n6xs krb5kdc[11246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:46:03.967690 11690 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:46:03.969208 11690 webserver.cc:460] Webserver started at http://127.0.97.62:42665/ using document root <none> and password file <none>
I20260928 08:46:03.969885 11690 fs_manager.cc:359] Metadata directory not provided
I20260928 08:46:03.970005 11690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:46:03.970317 11690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 08:46:03 dist-test-slave-n6xs krb5kdc[11246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585163, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 08:46:04.027186 11690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "a9d62245bd6b4e6f9bece7544bd549e2"
format_stamp: "Formatted at 2026-09-28 08:46:04 on dist-test-slave-n6xs"
server_key: "923d40e30b3fe98c20aef1136343c99e"
server_key_iv: "b3968f36e9df7b22706a382c70b482af"
server_key_version: "kuduclusterkey@0"
I20260928 08:46:04.028122 11690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "a9d62245bd6b4e6f9bece7544bd549e2"
format_stamp: "Formatted at 2026-09-28 08:46:04 on dist-test-slave-n6xs"
server_key: "923d40e30b3fe98c20aef1136343c99e"
server_key_iv: "b3968f36e9df7b22706a382c70b482af"
server_key_version: "kuduclusterkey@0"
I20260928 08:46:04.034114 11690 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260928 08:46:04.049855 11705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:46:04.051401 11690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:46:04.051589 11690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "a9d62245bd6b4e6f9bece7544bd549e2"
format_stamp: "Formatted at 2026-09-28 08:46:04 on dist-test-slave-n6xs"
server_key: "923d40e30b3fe98c20aef1136343c99e"
server_key_iv: "b3968f36e9df7b22706a382c70b482af"
server_key_version: "kuduclusterkey@0"
I20260928 08:46:04.051807 11690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:46:04.068742 11690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:46:04.073230 11690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:46:04.073515 11690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:46:04.093209 11690 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41113
I20260928 08:46:04.093218 11756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41113 every 8 connection(s)
I20260928 08:46:04.094509 11690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:46:04.096405 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 11690
I20260928 08:46:04.096621 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:46:04.097384 388 external_mini_cluster.cc:1428] Setting key b8176ac92115c3a60a84db394969e3b4
I20260928 08:46:04.099378 11757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:46:04.118212 11757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9d62245bd6b4e6f9bece7544bd549e2: Bootstrap starting.
I20260928 08:46:04.122020 11757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a9d62245bd6b4e6f9bece7544bd549e2: Neither blocks nor log segments found. Creating new log.
I20260928 08:46:04.123251 11757 log.cc:824] T 00000000000000000000000000000000 P a9d62245bd6b4e6f9bece7544bd549e2: Log is configured to *not* fsync() on all Append() calls
Sep 28 08:46:04 dist-test-slave-n6xs krb5kdc[11246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:46:04.126329 11757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a9d62245bd6b4e6f9bece7544bd549e2: No bootstrap required, opened a new log
I20260928 08:46:04.135327 11760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:04.099489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42182 (local address 127.0.97.62:41113)
0928 08:46:04.112080 (+ 12591us) server_negotiation.cc:206] Beginning negotiation
0928 08:46:04.112095 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 08:46:04.112146 (+ 51us) server_negotiation.cc:407] Connection header received
0928 08:46:04.114059 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:46:04.114095 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:46:04.114637 (+ 542us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:46:04.115089 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:46:04.116574 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:04.118449 (+ 1875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:04.120329 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:04.121768 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:04.121970 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:04.126733 (+ 4763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:46:04.126743 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:46:04.126764 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:46:04.126814 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:46:04.130610 (+ 3796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:46:04.131436 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:46:04.131447 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:46:04.131456 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:46:04.131641 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:46:04.132079 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:46:04.132084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:46:04.132088 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:46:04.132803 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:46:04.133101 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0928 08:46:04.133576 (+ 475us) server_negotiation.cc:299] Negotiation successful
0928 08:46:04.133968 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12348,"thread_start_us":12112,"threads_started":1}
I20260928 08:46:04.135085 11757 raft_consensus.cc:348] T 00000000000000000000000000000000 P a9d62245bd6b4e6f9bece7544bd549e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d62245bd6b4e6f9bece7544bd549e2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41113 } }
I20260928 08:46:04.135957 11757 raft_consensus.cc:374] T 00000000000000000000000000000000 P a9d62245bd6b4e6f9bece7544bd549e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:46:04.136080 11757 raft_consensus.cc:729] T 00000000000000000000000000000000 P a9d62245bd6b4e6f9bece7544bd549e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9d62245bd6b4e6f9bece7544bd549e2, State: Initialized, Role: FOLLOWER
I20260928 08:46:04.136741 11757 consensus_queue.cc:260] T 00000000000000000000000000000000 P a9d62245bd6b4e6f9bece7544bd549e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d62245bd6b4e6f9bece7544bd549e2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41113 } }
I20260928 08:46:04.137056 11757 raft_consensus.cc:388] T 00000000000000000000000000000000 P a9d62245bd6b4e6f9bece7544bd549e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:46:04.137225 11757 raft_consensus.cc:482] T 00000000000000000000000000000000 P a9d62245bd6b4e6f9bece7544bd549e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:46:04.137388 11757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a9d62245bd6b4e6f9bece7544bd549e2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:46:04.139844 11757 raft_consensus.cc:504] T 00000000000000000000000000000000 P a9d62245bd6b4e6f9bece7544bd549e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d62245bd6b4e6f9bece7544bd549e2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41113 } }
I20260928 08:46:04.140316 11757 leader_election.cc:302] T 00000000000000000000000000000000 P a9d62245bd6b4e6f9bece7544bd549e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a9d62245bd6b4e6f9bece7544bd549e2; no voters:
I20260928 08:46:04.140877 11757 leader_election.cc:288] T 00000000000000000000000000000000 P a9d62245bd6b4e6f9bece7544bd549e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:46:04.141004 11762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a9d62245bd6b4e6f9bece7544bd549e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:46:04.142024 11762 raft_consensus.cc:686] T 00000000000000000000000000000000 P a9d62245bd6b4e6f9bece7544bd549e2 [term 1 LEADER]: Becoming Leader. State: Replica: a9d62245bd6b4e6f9bece7544bd549e2, State: Running, Role: LEADER
I20260928 08:46:04.142659 11762 consensus_queue.cc:237] T 00000000000000000000000000000000 P a9d62245bd6b4e6f9bece7544bd549e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d62245bd6b4e6f9bece7544bd549e2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41113 } }
I20260928 08:46:04.142835 11757 sys_catalog.cc:564] T 00000000000000000000000000000000 P a9d62245bd6b4e6f9bece7544bd549e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:46:04.146001 11764 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9d62245bd6b4e6f9bece7544bd549e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a9d62245bd6b4e6f9bece7544bd549e2. Latest consensus state: current_term: 1 leader_uuid: "a9d62245bd6b4e6f9bece7544bd549e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d62245bd6b4e6f9bece7544bd549e2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41113 } } }
I20260928 08:46:04.146337 11764 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9d62245bd6b4e6f9bece7544bd549e2 [sys.catalog]: This master's current role is: LEADER
I20260928 08:46:04.147373 11763 sys_catalog.cc:455] T 00000000000000000000000000000000 P a9d62245bd6b4e6f9bece7544bd549e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a9d62245bd6b4e6f9bece7544bd549e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a9d62245bd6b4e6f9bece7544bd549e2" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41113 } } }
I20260928 08:46:04.147653 11763 sys_catalog.cc:458] T 00000000000000000000000000000000 P a9d62245bd6b4e6f9bece7544bd549e2 [sys.catalog]: This master's current role is: LEADER
I20260928 08:46:04.149596 11768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:46:04.155859 11768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:46:04.177934 11768 catalog_manager.cc:1347] Generated new cluster ID: 563e7be6af7b47309e2032c3a960e1d8
I20260928 08:46:04.178054 11768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:46:04.192891 11757 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 08:46:04.195964 11768 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:46:04.197759 11768 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:46:04.217391 11768 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a9d62245bd6b4e6f9bece7544bd549e2: Generated new TSK 0
I20260928 08:46:04.218633 11768 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:46:05.036358 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:42953/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41113
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:46:05 dist-test-slave-n6xs krb5kdc[11246](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790585165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 08:46:05.470986 11802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:46:05.471536 11802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:46:05.471766 11802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:46:05.483953 11802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:46:05.484117 11802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:46:05.484344 11802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:46:05.500649 11802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:42953/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:46:05.502447 11802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:46:05.504230 11802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:46:05.519245 11813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:46:05.519207 11815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:46:05.520232 11812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:46:05.521600 11802 server_base.cc:1034] running on GCE node
I20260928 08:46:05.522248 11802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:46:05.523406 11802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:46:05.526787 11802 hybrid_clock.cc:648] HybridClock initialized: now 1790585165526740 us; error 38 us; skew 500 ppm
I20260928 08:46:05.527155 11802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:46:05 dist-test-slave-n6xs krb5kdc[11246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:46:05.530740 11802 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:46:05.554603 11802 webserver.cc:460] Webserver started at http://127.0.97.1:41381/ using document root <none> and password file <none>
I20260928 08:46:05.555727 11802 fs_manager.cc:359] Metadata directory not provided
I20260928 08:46:05.555905 11802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:46:05.556316 11802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 08:46:05 dist-test-slave-n6xs krb5kdc[11246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 08:46:05.576730 11802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-0/data/instance:
uuid: "b2cc3f4599054772be08f0e661847282"
format_stamp: "Formatted at 2026-09-28 08:46:05 on dist-test-slave-n6xs"
server_key: "6f38a82a11d2c50f5ef38a46fd337d48"
server_key_iv: "933a6279ad481f9c57dd254787d64b53"
server_key_version: "kuduclusterkey@0"
I20260928 08:46:05.577941 11802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-0/wal/instance:
uuid: "b2cc3f4599054772be08f0e661847282"
format_stamp: "Formatted at 2026-09-28 08:46:05 on dist-test-slave-n6xs"
server_key: "6f38a82a11d2c50f5ef38a46fd337d48"
server_key_iv: "933a6279ad481f9c57dd254787d64b53"
server_key_version: "kuduclusterkey@0"
I20260928 08:46:05.603658 11802 fs_manager.cc:691] Time spent creating directory manager: real 0.025s user 0.012s sys 0.012s
I20260928 08:46:05.636477 11822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:46:05.638700 11802 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.004s sys 0.008s
I20260928 08:46:05.639003 11802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-0/wal
uuid: "b2cc3f4599054772be08f0e661847282"
format_stamp: "Formatted at 2026-09-28 08:46:05 on dist-test-slave-n6xs"
server_key: "6f38a82a11d2c50f5ef38a46fd337d48"
server_key_iv: "933a6279ad481f9c57dd254787d64b53"
server_key_version: "kuduclusterkey@0"
I20260928 08:46:05.639393 11802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:46:05.679237 11802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:46:05.684356 11802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:46:05.684718 11802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:46:05.689230 11802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:46:05.691109 11802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:46:05.691208 11802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:46:05.691425 11802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:46:05.691627 11802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 08:46:05 dist-test-slave-n6xs krb5kdc[11246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 08:46:05.900501 11802 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:36871
I20260928 08:46:05.900523 11934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:36871 every 8 connection(s)
I20260928 08:46:05.901842 11802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:46:05.905198 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 11802
I20260928 08:46:05.905392 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:46:05.906199 388 external_mini_cluster.cc:1428] Setting key 451282003bf8ef2574d9a06cd7195762
Sep 28 08:46:05 dist-test-slave-n6xs krb5kdc[11246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
I20260928 08:46:05.964231 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:05.906758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:50517 (local address 127.0.97.62:41113)
0928 08:46:05.927258 (+ 20500us) server_negotiation.cc:206] Beginning negotiation
0928 08:46:05.927268 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:46:05.931880 (+ 4612us) server_negotiation.cc:407] Connection header received
0928 08:46:05.936444 (+ 4564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:46:05.936450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:46:05.936558 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:46:05.936675 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:46:05.942240 (+ 5565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:05.943477 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:05.946213 (+ 2736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:05.946863 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:05.946994 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:05.951966 (+ 4972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:46:05.951977 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:46:05.951982 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:46:05.952044 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:46:05.955941 (+ 3897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:46:05.961307 (+ 5366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:46:05.961313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:46:05.961317 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:46:05.961503 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:46:05.962096 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:46:05.962102 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:46:05.962105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:46:05.962495 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:46:05.962623 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 08:46:05.963594 (+ 971us) server_negotiation.cc:299] Negotiation successful
0928 08:46:05.963864 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20288,"thread_start_us":20514,"threads_started":1}
I20260928 08:46:05.966442 11937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:05.905911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41113 (local address 127.0.97.1:50517)
0928 08:46:05.928175 (+ 22264us) negotiation.cc:107] Waiting for socket to connect
0928 08:46:05.928253 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0928 08:46:05.936106 (+ 7853us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:46:05.940284 (+ 4178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:46:05.940299 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:46:05.941014 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:46:05.941946 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:46:05.942034 (+ 88us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:46:05.943736 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:46:05.943741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:46:05.946022 (+ 2281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:46:05.946049 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:46:05.947360 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:46:05.947369 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:46:05.947668 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:05.948556 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:46:05.948587 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:46:05.951629 (+ 3042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:46:05.960009 (+ 8380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:46:05.960020 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:46:05.960041 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:46:05.961149 (+ 1108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:46:05.961658 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:46:05.961662 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:46:05.961666 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:46:05.961947 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:46:05.962714 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:46:05.962723 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:46:05.963164 (+ 441us) client_negotiation.cc:769] Sending connection context
0928 08:46:05.964537 (+ 1373us) client_negotiation.cc:240] Negotiation successful
0928 08:46:05.965090 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21792,"thread_start_us":25858,"threads_started":1}
I20260928 08:46:05.968525 11935 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41113
I20260928 08:46:05.968834 11935 heartbeater.cc:464] Registering TS with master...
I20260928 08:46:05.969656 11935 heartbeater.cc:511] Master 127.0.97.62:41113 requested a full tablet report, sending...
I20260928 08:46:05.972127 11722 ts_manager.cc:194] Registered new tserver with Master: b2cc3f4599054772be08f0e661847282 (127.0.97.1:36871)
I20260928 08:46:05.974198 11722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:50517
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 08:46:05.992324 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:42953/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41113
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:46:06 dist-test-slave-n6xs krb5kdc[11246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260928 08:46:06.452461 11941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:46:06.452780 11941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:46:06.452941 11941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:46:06.460704 11941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:46:06.460848 11941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:46:06.461016 11941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:46:06.471000 11941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:42953/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:46:06.472352 11941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:46:06.473650 11941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:46:06.488526 11956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:46:06.488636 11958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:46:06.488600 11955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:46:06.490080 11941 server_base.cc:1034] running on GCE node
I20260928 08:46:06.490720 11941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:46:06.491865 11941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:46:06.493129 11941 hybrid_clock.cc:648] HybridClock initialized: now 1790585166493036 us; error 76 us; skew 500 ppm
I20260928 08:46:06.493481 11941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:46:06 dist-test-slave-n6xs krb5kdc[11246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:46:06.497066 11941 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:46:06.499358 11941 webserver.cc:460] Webserver started at http://127.0.97.2:36727/ using document root <none> and password file <none>
I20260928 08:46:06.500109 11941 fs_manager.cc:359] Metadata directory not provided
I20260928 08:46:06.500324 11941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:46:06.500659 11941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 08:46:06 dist-test-slave-n6xs krb5kdc[11246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 08:46:06.519181 11941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-1/data/instance:
uuid: "82e8686fb2d14bcaafe6c1d005f098bf"
format_stamp: "Formatted at 2026-09-28 08:46:06 on dist-test-slave-n6xs"
server_key: "344a1b23f50cf5af5c60953475727986"
server_key_iv: "d10bf4349e657dd44c6ad3a221307b9f"
server_key_version: "kuduclusterkey@0"
I20260928 08:46:06.520545 11941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-1/wal/instance:
uuid: "82e8686fb2d14bcaafe6c1d005f098bf"
format_stamp: "Formatted at 2026-09-28 08:46:06 on dist-test-slave-n6xs"
server_key: "344a1b23f50cf5af5c60953475727986"
server_key_iv: "d10bf4349e657dd44c6ad3a221307b9f"
server_key_version: "kuduclusterkey@0"
I20260928 08:46:06.528178 11941 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260928 08:46:06.556425 11965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:46:06.558240 11941 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.006s sys 0.006s
I20260928 08:46:06.558477 11941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-1/wal
uuid: "82e8686fb2d14bcaafe6c1d005f098bf"
format_stamp: "Formatted at 2026-09-28 08:46:06 on dist-test-slave-n6xs"
server_key: "344a1b23f50cf5af5c60953475727986"
server_key_iv: "d10bf4349e657dd44c6ad3a221307b9f"
server_key_version: "kuduclusterkey@0"
I20260928 08:46:06.558732 11941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:46:06.583398 11941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:46:06.587777 11941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:46:06.588205 11941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:46:06.589293 11941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:46:06.591240 11941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:46:06.591365 11941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:46:06.591526 11941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:46:06.591600 11941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:46:06.648612 11941 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:37281
I20260928 08:46:06.648630 12077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:37281 every 8 connection(s)
I20260928 08:46:06.650179 11941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:46:06.656203 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 11941
I20260928 08:46:06.656450 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:46:06.657225 388 external_mini_cluster.cc:1428] Setting key 1e603109df26df85764abf1e5f5853ac
Sep 28 08:46:06 dist-test-slave-n6xs krb5kdc[11246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585166, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260928 08:46:06.692806 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:06.653691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:36423 (local address 127.0.97.62:41113)
0928 08:46:06.673804 (+ 20113us) server_negotiation.cc:206] Beginning negotiation
0928 08:46:06.673814 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:46:06.673856 (+ 42us) server_negotiation.cc:407] Connection header received
0928 08:46:06.673919 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:46:06.673923 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:46:06.674006 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:46:06.674169 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:46:06.676805 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:06.677572 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:06.680056 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:06.680484 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:06.680649 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:06.685956 (+ 5307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:46:06.685962 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:46:06.685966 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:46:06.686011 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:46:06.688852 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:46:06.689797 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:46:06.689811 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:46:06.689814 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:46:06.689951 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:46:06.690531 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:46:06.690539 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:46:06.690549 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:46:06.690834 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:46:06.691052 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 08:46:06.692125 (+ 1073us) server_negotiation.cc:299] Negotiation successful
0928 08:46:06.692392 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19935,"thread_start_us":20093,"threads_started":1}
I20260928 08:46:06.694329 12081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:06.654700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41113 (local address 127.0.97.2:36423)
0928 08:46:06.660418 (+ 5718us) negotiation.cc:107] Waiting for socket to connect
0928 08:46:06.660488 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 08:46:06.662918 (+ 2430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:46:06.674469 (+ 11551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:46:06.674482 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:46:06.675640 (+ 1158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:46:06.676587 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:46:06.676617 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:46:06.677749 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:46:06.677755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:46:06.679867 (+ 2112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:46:06.679896 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:46:06.681529 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:46:06.681537 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:46:06.681846 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:06.682716 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:46:06.682749 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:46:06.685682 (+ 2933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:46:06.689032 (+ 3350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:46:06.689042 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:46:06.689061 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:46:06.689669 (+ 608us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:46:06.690068 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:46:06.690075 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:46:06.690078 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:46:06.690408 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:46:06.691024 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:46:06.691033 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:46:06.691563 (+ 530us) client_negotiation.cc:769] Sending connection context
0928 08:46:06.691979 (+ 416us) client_negotiation.cc:240] Negotiation successful
0928 08:46:06.692546 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5382,"thread_start_us":12983,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260928 08:46:06.696517 12078 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41113
I20260928 08:46:06.696856 12078 heartbeater.cc:464] Registering TS with master...
I20260928 08:46:06.697747 12078 heartbeater.cc:511] Master 127.0.97.62:41113 requested a full tablet report, sending...
I20260928 08:46:06.699496 11722 ts_manager.cc:194] Registered new tserver with Master: 82e8686fb2d14bcaafe6c1d005f098bf (127.0.97.2:37281)
I20260928 08:46:06.700697 11722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:36423
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:46:06.731210 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:42953/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41113
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:46:06.978184 11935 heartbeater.cc:503] Master 127.0.97.62:41113 was elected leader, sending a full tablet report...
W20260928 08:46:07.074892 12084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:46:07.075191 12084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:46:07.075366 12084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:46:07.082803 12084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:46:07.082924 12084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:46:07.083180 12084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:46:07.093192 12084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:42953/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41113
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:46:07.094583 12084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:46:07.096012 12084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:46:07.110041 12089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:46:07.110136 12090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:46:07.110086 12092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:46:07.113492 12084 server_base.cc:1034] running on GCE node
I20260928 08:46:07.114141 12084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:46:07.115134 12084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:46:07.116389 12084 hybrid_clock.cc:648] HybridClock initialized: now 1790585167116331 us; error 34 us; skew 500 ppm
I20260928 08:46:07.116720 12084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:46:07 dist-test-slave-n6xs krb5kdc[11246](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:46:07.120160 12084 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:46:07.121726 12084 webserver.cc:460] Webserver started at http://127.0.97.3:46505/ using document root <none> and password file <none>
I20260928 08:46:07.122486 12084 fs_manager.cc:359] Metadata directory not provided
I20260928 08:46:07.122638 12084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:46:07.122983 12084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 08:46:07 dist-test-slave-n6xs krb5kdc[11246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 08:46:07.137981 12084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-2/data/instance:
uuid: "d6f64b14e16e4b11bb7afdd7811dfc38"
format_stamp: "Formatted at 2026-09-28 08:46:07 on dist-test-slave-n6xs"
server_key: "3558e9a25f055987cc15ffa5f7e7a225"
server_key_iv: "5dbf2d1ee07fc82e25328f35cff9d2ce"
server_key_version: "kuduclusterkey@0"
I20260928 08:46:07.138872 12084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d6f64b14e16e4b11bb7afdd7811dfc38"
format_stamp: "Formatted at 2026-09-28 08:46:07 on dist-test-slave-n6xs"
server_key: "3558e9a25f055987cc15ffa5f7e7a225"
server_key_iv: "5dbf2d1ee07fc82e25328f35cff9d2ce"
server_key_version: "kuduclusterkey@0"
I20260928 08:46:07.144802 12084 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 08:46:07.158761 12099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:46:07.160467 12084 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 08:46:07.160650 12084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-2/wal
uuid: "d6f64b14e16e4b11bb7afdd7811dfc38"
format_stamp: "Formatted at 2026-09-28 08:46:07 on dist-test-slave-n6xs"
server_key: "3558e9a25f055987cc15ffa5f7e7a225"
server_key_iv: "5dbf2d1ee07fc82e25328f35cff9d2ce"
server_key_version: "kuduclusterkey@0"
I20260928 08:46:07.160907 12084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:46:07.177644 12084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:46:07.182196 12084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:46:07.182492 12084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:46:07.183404 12084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:46:07.184798 12084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:46:07.184907 12084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:46:07.185050 12084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:46:07.185120 12084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:46:07.220474 12084 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:37457
I20260928 08:46:07.220530 12211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:37457 every 8 connection(s)
I20260928 08:46:07.221836 12084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:46:07.223584 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 12084
I20260928 08:46:07.223752 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:46:07.224242 388 external_mini_cluster.cc:1428] Setting key 1f72c388752f73ade63fd58fddcd880f
Sep 28 08:46:07 dist-test-slave-n6xs krb5kdc[11246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585167, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:46:07.242425 12214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:07.224766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36437 (local address 127.0.97.62:41113)
0928 08:46:07.225397 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0928 08:46:07.225403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:46:07.226965 (+ 1562us) server_negotiation.cc:407] Connection header received
0928 08:46:07.228312 (+ 1347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:46:07.228316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:46:07.228411 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:46:07.228507 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:46:07.230250 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:07.231069 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:07.232678 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:07.233080 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:07.233182 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:07.236475 (+ 3293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:46:07.236480 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:46:07.236483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:46:07.236513 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:46:07.239711 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:46:07.240403 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:46:07.240407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:46:07.240410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:46:07.240527 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:46:07.240976 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:46:07.240980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:46:07.240983 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:46:07.241241 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:46:07.241347 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 08:46:07.242009 (+ 662us) server_negotiation.cc:299] Negotiation successful
0928 08:46:07.242160 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":391,"threads_started":1}
I20260928 08:46:07.243456 12215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:07.225353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41113 (local address 127.0.97.3:36437)
0928 08:46:07.226197 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0928 08:46:07.226241 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:46:07.228063 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:46:07.228700 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:46:07.228708 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:46:07.229112 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:46:07.230086 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:46:07.230104 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:46:07.231205 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:46:07.231208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:46:07.232518 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:46:07.232536 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:46:07.233266 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:46:07.233271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:46:07.233478 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:07.234158 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:46:07.234179 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:46:07.236247 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:46:07.239854 (+ 3607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:46:07.239863 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:46:07.239880 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:46:07.240279 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:46:07.240649 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:46:07.240652 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:46:07.240655 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:46:07.240873 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:46:07.241354 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:46:07.241360 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:46:07.241655 (+ 295us) client_negotiation.cc:769] Sending connection context
0928 08:46:07.241928 (+ 273us) client_negotiation.cc:240] Negotiation successful
0928 08:46:07.242291 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":374,"threads_started":1}
I20260928 08:46:07.244979 12212 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41113
I20260928 08:46:07.245298 12212 heartbeater.cc:464] Registering TS with master...
I20260928 08:46:07.245972 12212 heartbeater.cc:511] Master 127.0.97.62:41113 requested a full tablet report, sending...
I20260928 08:46:07.247390 11722 ts_manager.cc:194] Registered new tserver with Master: d6f64b14e16e4b11bb7afdd7811dfc38 (127.0.97.3:37457)
I20260928 08:46:07.248216 11722 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:36437
I20260928 08:46:07.251959 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:46:07.269464 12214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:07.256363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42194 (local address 127.0.97.62:41113)
0928 08:46:07.256593 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 08:46:07.256603 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:46:07.257190 (+ 587us) server_negotiation.cc:407] Connection header received
0928 08:46:07.257308 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:46:07.257312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:46:07.257388 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:46:07.257473 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:46:07.259144 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:07.259897 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:07.261288 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:07.261675 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:07.261763 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:07.264356 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:46:07.264360 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:46:07.264364 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:46:07.264414 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:46:07.267093 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:46:07.267699 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:46:07.267702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:46:07.267705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:46:07.267850 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:46:07.268259 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:46:07.268263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:46:07.268265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:46:07.268567 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:46:07.268752 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 08:46:07.269074 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 08:46:07.269235 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 08:46:07.273476 11722 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:46:07.280903 11722 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:46:07.305786 12226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:07.293964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41744 (local address 127.0.97.1:36871)
0928 08:46:07.296101 (+ 2137us) server_negotiation.cc:206] Beginning negotiation
0928 08:46:07.296110 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:46:07.296139 (+ 29us) server_negotiation.cc:407] Connection header received
0928 08:46:07.296234 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:46:07.296242 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:46:07.296521 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:46:07.296739 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:46:07.297753 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:07.299724 (+ 1971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:07.302608 (+ 2884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:07.304178 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:07.304330 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:07.304559 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 08:46:07.305210 (+ 651us) server_negotiation.cc:299] Negotiation successful
0928 08:46:07.305374 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1976,"thread_start_us":1832,"threads_started":1}
I20260928 08:46:07.306579 12227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:07.293764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:36871 (local address 127.0.0.1:41744)
0928 08:46:07.295157 (+ 1393us) negotiation.cc:107] Waiting for socket to connect
0928 08:46:07.295210 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 08:46:07.295429 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:46:07.297042 (+ 1613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:46:07.297051 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:46:07.297091 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:46:07.297570 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:46:07.297581 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:46:07.299908 (+ 2327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:46:07.299915 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:46:07.302433 (+ 2518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:46:07.302451 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:46:07.304692 (+ 2241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:46:07.304698 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:46:07.304872 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:07.304923 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 08:46:07.305048 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 08:46:07.306145 (+ 1097us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1075,"mutex_wait_us":892,"thread_start_us":3037,"threads_started":1}
I20260928 08:46:07.309331 12225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:07.293981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37457 (local address 127.0.0.1:36774)
0928 08:46:07.297068 (+ 3087us) negotiation.cc:107] Waiting for socket to connect
0928 08:46:07.297086 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:46:07.297209 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:46:07.299680 (+ 2471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:46:07.299684 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:46:07.299708 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:46:07.300195 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:46:07.300211 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:46:07.303464 (+ 3253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:46:07.303468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:46:07.305596 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:46:07.305611 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:46:07.308510 (+ 2899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:46:07.308517 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:46:07.308713 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:07.308747 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 08:46:07.308834 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 08:46:07.308967 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2969,"mutex_wait_us":1,"thread_start_us":1007,"threads_started":1}
I20260928 08:46:07.309675 12228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:07.294698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36774 (local address 127.0.97.3:37457)
0928 08:46:07.297752 (+ 3054us) server_negotiation.cc:206] Beginning negotiation
0928 08:46:07.297760 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:46:07.297789 (+ 29us) server_negotiation.cc:407] Connection header received
0928 08:46:07.298301 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:46:07.298310 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:46:07.298539 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:46:07.300412 (+ 1873us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:46:07.300676 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:07.303323 (+ 2647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:07.305803 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:07.308279 (+ 2476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:07.308440 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:07.308710 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 08:46:07.308927 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 08:46:07.309096 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2874,"thread_start_us":2981,"threads_started":1}
I20260928 08:46:07.312175 11870 tablet_service.cc:1467] Processing CreateTablet for tablet d2205cf858334d95aca2f2866ccce3ea (DEFAULT_TABLE table=test-table [id=c86aa0f0816d499abd326ad4148d4605]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:46:07.313740 11870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2205cf858334d95aca2f2866ccce3ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:46:07.313690 12147 tablet_service.cc:1467] Processing CreateTablet for tablet d2205cf858334d95aca2f2866ccce3ea (DEFAULT_TABLE table=test-table [id=c86aa0f0816d499abd326ad4148d4605]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:46:07.314953 12227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:07.297283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37281 (local address 127.0.0.1:33506)
0928 08:46:07.306785 (+ 9502us) negotiation.cc:107] Waiting for socket to connect
0928 08:46:07.306804 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:46:07.306907 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:46:07.308803 (+ 1896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:46:07.308839 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:46:07.308871 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:46:07.309024 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:46:07.309036 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:46:07.311203 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:46:07.311213 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:46:07.312799 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:46:07.312811 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:46:07.314120 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:46:07.314125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:46:07.314385 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:07.314419 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 08:46:07.314517 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 08:46:07.314658 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9402,"thread_start_us":10855,"threads_started":1}
I20260928 08:46:07.315073 12230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:07.297492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33506 (local address 127.0.97.2:37281)
0928 08:46:07.307962 (+ 10470us) server_negotiation.cc:206] Beginning negotiation
0928 08:46:07.307971 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:46:07.307999 (+ 28us) server_negotiation.cc:407] Connection header received
0928 08:46:07.308089 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:46:07.308096 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:46:07.308357 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:46:07.308551 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:46:07.309190 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:07.311048 (+ 1858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:07.313012 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:07.313971 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:07.314111 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:07.314289 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 08:46:07.314576 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 08:46:07.314726 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10291,"thread_start_us":10183,"threads_started":1}
I20260928 08:46:07.315088 12147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2205cf858334d95aca2f2866ccce3ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:46:07.318184 12013 tablet_service.cc:1467] Processing CreateTablet for tablet d2205cf858334d95aca2f2866ccce3ea (DEFAULT_TABLE table=test-table [id=c86aa0f0816d499abd326ad4148d4605]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:46:07.319336 12013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2205cf858334d95aca2f2866ccce3ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:46:07.328577 12231 tablet_bootstrap.cc:492] T d2205cf858334d95aca2f2866ccce3ea P b2cc3f4599054772be08f0e661847282: Bootstrap starting.
I20260928 08:46:07.330847 12232 tablet_bootstrap.cc:492] T d2205cf858334d95aca2f2866ccce3ea P d6f64b14e16e4b11bb7afdd7811dfc38: Bootstrap starting.
I20260928 08:46:07.331710 12233 tablet_bootstrap.cc:492] T d2205cf858334d95aca2f2866ccce3ea P 82e8686fb2d14bcaafe6c1d005f098bf: Bootstrap starting.
I20260928 08:46:07.334303 12231 tablet_bootstrap.cc:654] T d2205cf858334d95aca2f2866ccce3ea P b2cc3f4599054772be08f0e661847282: Neither blocks nor log segments found. Creating new log.
I20260928 08:46:07.334635 12232 tablet_bootstrap.cc:654] T d2205cf858334d95aca2f2866ccce3ea P d6f64b14e16e4b11bb7afdd7811dfc38: Neither blocks nor log segments found. Creating new log.
I20260928 08:46:07.334973 12233 tablet_bootstrap.cc:654] T d2205cf858334d95aca2f2866ccce3ea P 82e8686fb2d14bcaafe6c1d005f098bf: Neither blocks nor log segments found. Creating new log.
I20260928 08:46:07.335378 12231 log.cc:824] T d2205cf858334d95aca2f2866ccce3ea P b2cc3f4599054772be08f0e661847282: Log is configured to *not* fsync() on all Append() calls
I20260928 08:46:07.336236 12232 log.cc:824] T d2205cf858334d95aca2f2866ccce3ea P d6f64b14e16e4b11bb7afdd7811dfc38: Log is configured to *not* fsync() on all Append() calls
I20260928 08:46:07.336541 12233 log.cc:824] T d2205cf858334d95aca2f2866ccce3ea P 82e8686fb2d14bcaafe6c1d005f098bf: Log is configured to *not* fsync() on all Append() calls
I20260928 08:46:07.338312 12231 tablet_bootstrap.cc:492] T d2205cf858334d95aca2f2866ccce3ea P b2cc3f4599054772be08f0e661847282: No bootstrap required, opened a new log
I20260928 08:46:07.338581 12231 ts_tablet_manager.cc:1397] T d2205cf858334d95aca2f2866ccce3ea P b2cc3f4599054772be08f0e661847282: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.002s
I20260928 08:46:07.339304 12233 tablet_bootstrap.cc:492] T d2205cf858334d95aca2f2866ccce3ea P 82e8686fb2d14bcaafe6c1d005f098bf: No bootstrap required, opened a new log
I20260928 08:46:07.339551 12233 ts_tablet_manager.cc:1397] T d2205cf858334d95aca2f2866ccce3ea P 82e8686fb2d14bcaafe6c1d005f098bf: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260928 08:46:07.339696 12232 tablet_bootstrap.cc:492] T d2205cf858334d95aca2f2866ccce3ea P d6f64b14e16e4b11bb7afdd7811dfc38: No bootstrap required, opened a new log
I20260928 08:46:07.339954 12232 ts_tablet_manager.cc:1397] T d2205cf858334d95aca2f2866ccce3ea P d6f64b14e16e4b11bb7afdd7811dfc38: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260928 08:46:07.347429 12233 raft_consensus.cc:348] T d2205cf858334d95aca2f2866ccce3ea P 82e8686fb2d14bcaafe6c1d005f098bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2cc3f4599054772be08f0e661847282" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36871 } } peers { permanent_uuid: "d6f64b14e16e4b11bb7afdd7811dfc38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37457 } } peers { permanent_uuid: "82e8686fb2d14bcaafe6c1d005f098bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37281 } }
I20260928 08:46:07.347810 12233 raft_consensus.cc:374] T d2205cf858334d95aca2f2866ccce3ea P 82e8686fb2d14bcaafe6c1d005f098bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:46:07.347939 12233 raft_consensus.cc:729] T d2205cf858334d95aca2f2866ccce3ea P 82e8686fb2d14bcaafe6c1d005f098bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82e8686fb2d14bcaafe6c1d005f098bf, State: Initialized, Role: FOLLOWER
I20260928 08:46:07.348552 12233 consensus_queue.cc:260] T d2205cf858334d95aca2f2866ccce3ea P 82e8686fb2d14bcaafe6c1d005f098bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2cc3f4599054772be08f0e661847282" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36871 } } peers { permanent_uuid: "d6f64b14e16e4b11bb7afdd7811dfc38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37457 } } peers { permanent_uuid: "82e8686fb2d14bcaafe6c1d005f098bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37281 } }
I20260928 08:46:07.349864 12078 heartbeater.cc:503] Master 127.0.97.62:41113 was elected leader, sending a full tablet report...
I20260928 08:46:07.349514 12231 raft_consensus.cc:348] T d2205cf858334d95aca2f2866ccce3ea P b2cc3f4599054772be08f0e661847282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2cc3f4599054772be08f0e661847282" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36871 } } peers { permanent_uuid: "d6f64b14e16e4b11bb7afdd7811dfc38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37457 } } peers { permanent_uuid: "82e8686fb2d14bcaafe6c1d005f098bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37281 } }
I20260928 08:46:07.350555 12231 raft_consensus.cc:374] T d2205cf858334d95aca2f2866ccce3ea P b2cc3f4599054772be08f0e661847282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:46:07.350703 12231 raft_consensus.cc:729] T d2205cf858334d95aca2f2866ccce3ea P b2cc3f4599054772be08f0e661847282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2cc3f4599054772be08f0e661847282, State: Initialized, Role: FOLLOWER
I20260928 08:46:07.351018 12232 raft_consensus.cc:348] T d2205cf858334d95aca2f2866ccce3ea P d6f64b14e16e4b11bb7afdd7811dfc38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2cc3f4599054772be08f0e661847282" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36871 } } peers { permanent_uuid: "d6f64b14e16e4b11bb7afdd7811dfc38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37457 } } peers { permanent_uuid: "82e8686fb2d14bcaafe6c1d005f098bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37281 } }
I20260928 08:46:07.351424 12232 raft_consensus.cc:374] T d2205cf858334d95aca2f2866ccce3ea P d6f64b14e16e4b11bb7afdd7811dfc38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:46:07.351511 12232 raft_consensus.cc:729] T d2205cf858334d95aca2f2866ccce3ea P d6f64b14e16e4b11bb7afdd7811dfc38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6f64b14e16e4b11bb7afdd7811dfc38, State: Initialized, Role: FOLLOWER
I20260928 08:46:07.352111 12232 consensus_queue.cc:260] T d2205cf858334d95aca2f2866ccce3ea P d6f64b14e16e4b11bb7afdd7811dfc38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2cc3f4599054772be08f0e661847282" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36871 } } peers { permanent_uuid: "d6f64b14e16e4b11bb7afdd7811dfc38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37457 } } peers { permanent_uuid: "82e8686fb2d14bcaafe6c1d005f098bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37281 } }
I20260928 08:46:07.352075 12231 consensus_queue.cc:260] T d2205cf858334d95aca2f2866ccce3ea P b2cc3f4599054772be08f0e661847282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2cc3f4599054772be08f0e661847282" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36871 } } peers { permanent_uuid: "d6f64b14e16e4b11bb7afdd7811dfc38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37457 } } peers { permanent_uuid: "82e8686fb2d14bcaafe6c1d005f098bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37281 } }
I20260928 08:46:07.353832 12233 ts_tablet_manager.cc:1428] T d2205cf858334d95aca2f2866ccce3ea P 82e8686fb2d14bcaafe6c1d005f098bf: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260928 08:46:07.355525 12232 ts_tablet_manager.cc:1428] T d2205cf858334d95aca2f2866ccce3ea P d6f64b14e16e4b11bb7afdd7811dfc38: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260928 08:46:07.355693 12231 ts_tablet_manager.cc:1428] T d2205cf858334d95aca2f2866ccce3ea P b2cc3f4599054772be08f0e661847282: Time spent starting tablet: real 0.017s user 0.007s sys 0.010s
I20260928 08:46:07.356352 12212 heartbeater.cc:503] Master 127.0.97.62:41113 was elected leader, sending a full tablet report...
I20260928 08:46:07.411859 12239 raft_consensus.cc:482] T d2205cf858334d95aca2f2866ccce3ea P b2cc3f4599054772be08f0e661847282 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:46:07.412264 12239 raft_consensus.cc:504] T d2205cf858334d95aca2f2866ccce3ea P b2cc3f4599054772be08f0e661847282 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2cc3f4599054772be08f0e661847282" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36871 } } peers { permanent_uuid: "d6f64b14e16e4b11bb7afdd7811dfc38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37457 } } peers { permanent_uuid: "82e8686fb2d14bcaafe6c1d005f098bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37281 } }
I20260928 08:46:07.414649 12239 leader_election.cc:288] T d2205cf858334d95aca2f2866ccce3ea P b2cc3f4599054772be08f0e661847282 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6f64b14e16e4b11bb7afdd7811dfc38 (127.0.97.3:37457), 82e8686fb2d14bcaafe6c1d005f098bf (127.0.97.2:37281)
I20260928 08:46:07.423113 12228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:07.415071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56449 (local address 127.0.97.3:37457)
0928 08:46:07.415278 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 08:46:07.415285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:46:07.416296 (+ 1011us) server_negotiation.cc:407] Connection header received
0928 08:46:07.416635 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:46:07.416641 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:46:07.416728 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:46:07.416792 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:46:07.418130 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:07.419407 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:07.421232 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:07.422039 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:07.422132 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:07.422203 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 08:46:07.422726 (+ 523us) server_negotiation.cc:299] Negotiation successful
0928 08:46:07.422848 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 08:46:07.423141 12230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:07.415245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34305 (local address 127.0.97.2:37281)
0928 08:46:07.415543 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0928 08:46:07.415547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:46:07.416946 (+ 1399us) server_negotiation.cc:407] Connection header received
0928 08:46:07.417059 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:46:07.417064 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:46:07.417172 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:46:07.417269 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:46:07.418676 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:07.419536 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:07.421308 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:07.422150 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:07.422306 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:07.422399 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 08:46:07.422762 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 08:46:07.422886 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159}
I20260928 08:46:07.423303 12240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:07.415172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:37457 (local address 127.0.97.1:56449)
0928 08:46:07.416170 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0928 08:46:07.416192 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:46:07.416436 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:46:07.417297 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:46:07.417300 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:46:07.417328 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:46:07.417863 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:46:07.417885 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:46:07.419542 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:46:07.419546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:46:07.421061 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:46:07.421072 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:46:07.422270 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:46:07.422277 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:46:07.422564 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:07.422599 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 08:46:07.422672 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 08:46:07.422814 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"mutex_wait_us":107,"thread_start_us":538,"threads_started":1}
I20260928 08:46:07.423281 12241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:07.415988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:37281 (local address 127.0.97.1:34305)
0928 08:46:07.416829 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0928 08:46:07.416846 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:46:07.416951 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:46:07.417456 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:46:07.417460 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:46:07.417488 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:46:07.418479 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:46:07.418494 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:46:07.419655 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:46:07.419658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:46:07.421156 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:46:07.421170 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:46:07.422383 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:46:07.422393 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:46:07.422576 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:07.422611 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 08:46:07.422717 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 08:46:07.422840 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":110,"thread_start_us":471,"threads_started":1}
W20260928 08:46:07.424418 12079 tablet.cc:2369] T d2205cf858334d95aca2f2866ccce3ea P 82e8686fb2d14bcaafe6c1d005f098bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:46:07.424407 12167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2205cf858334d95aca2f2866ccce3ea" candidate_uuid: "b2cc3f4599054772be08f0e661847282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6f64b14e16e4b11bb7afdd7811dfc38" is_pre_election: true
I20260928 08:46:07.424425 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2205cf858334d95aca2f2866ccce3ea" candidate_uuid: "b2cc3f4599054772be08f0e661847282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82e8686fb2d14bcaafe6c1d005f098bf" is_pre_election: true
I20260928 08:46:07.424968 12167 raft_consensus.cc:2454] T d2205cf858334d95aca2f2866ccce3ea P d6f64b14e16e4b11bb7afdd7811dfc38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2cc3f4599054772be08f0e661847282 in term 0.
I20260928 08:46:07.424970 12033 raft_consensus.cc:2454] T d2205cf858334d95aca2f2866ccce3ea P 82e8686fb2d14bcaafe6c1d005f098bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2cc3f4599054772be08f0e661847282 in term 0.
I20260928 08:46:07.425683 11826 leader_election.cc:302] T d2205cf858334d95aca2f2866ccce3ea P b2cc3f4599054772be08f0e661847282 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2cc3f4599054772be08f0e661847282, d6f64b14e16e4b11bb7afdd7811dfc38; no voters:
I20260928 08:46:07.426121 12239 raft_consensus.cc:2781] T d2205cf858334d95aca2f2866ccce3ea P b2cc3f4599054772be08f0e661847282 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:46:07.426268 12239 raft_consensus.cc:482] T d2205cf858334d95aca2f2866ccce3ea P b2cc3f4599054772be08f0e661847282 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:46:07.426368 12239 raft_consensus.cc:3037] T d2205cf858334d95aca2f2866ccce3ea P b2cc3f4599054772be08f0e661847282 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:46:07.429086 12239 raft_consensus.cc:504] T d2205cf858334d95aca2f2866ccce3ea P b2cc3f4599054772be08f0e661847282 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2cc3f4599054772be08f0e661847282" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36871 } } peers { permanent_uuid: "d6f64b14e16e4b11bb7afdd7811dfc38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37457 } } peers { permanent_uuid: "82e8686fb2d14bcaafe6c1d005f098bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37281 } }
I20260928 08:46:07.429672 12239 leader_election.cc:288] T d2205cf858334d95aca2f2866ccce3ea P b2cc3f4599054772be08f0e661847282 [CANDIDATE]: Term 1 election: Requested vote from peers d6f64b14e16e4b11bb7afdd7811dfc38 (127.0.97.3:37457), 82e8686fb2d14bcaafe6c1d005f098bf (127.0.97.2:37281)
I20260928 08:46:07.430200 12167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2205cf858334d95aca2f2866ccce3ea" candidate_uuid: "b2cc3f4599054772be08f0e661847282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6f64b14e16e4b11bb7afdd7811dfc38"
I20260928 08:46:07.430200 12033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2205cf858334d95aca2f2866ccce3ea" candidate_uuid: "b2cc3f4599054772be08f0e661847282" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82e8686fb2d14bcaafe6c1d005f098bf"
I20260928 08:46:07.430423 12033 raft_consensus.cc:3037] T d2205cf858334d95aca2f2866ccce3ea P 82e8686fb2d14bcaafe6c1d005f098bf [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:46:07.430406 12167 raft_consensus.cc:3037] T d2205cf858334d95aca2f2866ccce3ea P d6f64b14e16e4b11bb7afdd7811dfc38 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:46:07.433604 12033 raft_consensus.cc:2454] T d2205cf858334d95aca2f2866ccce3ea P 82e8686fb2d14bcaafe6c1d005f098bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2cc3f4599054772be08f0e661847282 in term 1.
I20260928 08:46:07.433602 12167 raft_consensus.cc:2454] T d2205cf858334d95aca2f2866ccce3ea P d6f64b14e16e4b11bb7afdd7811dfc38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2cc3f4599054772be08f0e661847282 in term 1.
I20260928 08:46:07.434017 11826 leader_election.cc:302] T d2205cf858334d95aca2f2866ccce3ea P b2cc3f4599054772be08f0e661847282 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2cc3f4599054772be08f0e661847282, d6f64b14e16e4b11bb7afdd7811dfc38; no voters:
I20260928 08:46:07.434295 12239 raft_consensus.cc:2781] T d2205cf858334d95aca2f2866ccce3ea P b2cc3f4599054772be08f0e661847282 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:46:07.434782 12239 raft_consensus.cc:686] T d2205cf858334d95aca2f2866ccce3ea P b2cc3f4599054772be08f0e661847282 [term 1 LEADER]: Becoming Leader. State: Replica: b2cc3f4599054772be08f0e661847282, State: Running, Role: LEADER
I20260928 08:46:07.435251 12239 consensus_queue.cc:237] T d2205cf858334d95aca2f2866ccce3ea P b2cc3f4599054772be08f0e661847282 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2cc3f4599054772be08f0e661847282" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36871 } } peers { permanent_uuid: "d6f64b14e16e4b11bb7afdd7811dfc38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37457 } } peers { permanent_uuid: "82e8686fb2d14bcaafe6c1d005f098bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37281 } }
W20260928 08:46:07.437179 11936 tablet.cc:2369] T d2205cf858334d95aca2f2866ccce3ea P b2cc3f4599054772be08f0e661847282: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:46:07.440208 11721 catalog_manager.cc:5495] T d2205cf858334d95aca2f2866ccce3ea P b2cc3f4599054772be08f0e661847282 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2cc3f4599054772be08f0e661847282 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "b2cc3f4599054772be08f0e661847282" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2cc3f4599054772be08f0e661847282" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 36871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6f64b14e16e4b11bb7afdd7811dfc38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37457 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82e8686fb2d14bcaafe6c1d005f098bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37281 } health_report { overall_health: UNKNOWN } } }
I20260928 08:46:07.461163 12226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:07.455678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41758 (local address 127.0.97.1:36871)
0928 08:46:07.455937 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 08:46:07.455942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:46:07.456020 (+ 78us) server_negotiation.cc:407] Connection header received
0928 08:46:07.456206 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:46:07.456210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:46:07.456284 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:46:07.456459 (+ 175us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:46:07.456850 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:07.457648 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:07.459016 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:07.459410 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:07.459543 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:07.460091 (+ 548us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:46:07.460597 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:46:07.460758 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 08:46:07.460809 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 08:46:07.460898 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 08:46:07.470060 12167 raft_consensus.cc:1261] T d2205cf858334d95aca2f2866ccce3ea P d6f64b14e16e4b11bb7afdd7811dfc38 [term 1 FOLLOWER]: Refusing update from remote peer b2cc3f4599054772be08f0e661847282: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:46:07.470058 12033 raft_consensus.cc:1261] T d2205cf858334d95aca2f2866ccce3ea P 82e8686fb2d14bcaafe6c1d005f098bf [term 1 FOLLOWER]: Refusing update from remote peer b2cc3f4599054772be08f0e661847282: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:46:07.471125 12242 consensus_queue.cc:1035] T d2205cf858334d95aca2f2866ccce3ea P b2cc3f4599054772be08f0e661847282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6f64b14e16e4b11bb7afdd7811dfc38" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 37457 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:46:07.471465 12239 consensus_queue.cc:1035] T d2205cf858334d95aca2f2866ccce3ea P b2cc3f4599054772be08f0e661847282 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82e8686fb2d14bcaafe6c1d005f098bf" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 37281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 08:46:07.473414 12213 tablet.cc:2369] T d2205cf858334d95aca2f2866ccce3ea P d6f64b14e16e4b11bb7afdd7811dfc38: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:46:07.482558 12246 mvcc.cc:205] Tried to move back new op lower bound from 7334236845946257408 to 7334236845818748928. Current Snapshot: MvccSnapshot[applied={T|T < 7334236845946257408}]
I20260928 08:46:07.484728 12245 mvcc.cc:205] Tried to move back new op lower bound from 7334236845946257408 to 7334236845818748928. Current Snapshot: MvccSnapshot[applied={T|T < 7334236845946257408}]
I20260928 08:46:07.498733 11721 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:46:07.498999 11721 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:46:07.502238 11721 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a9d62245bd6b4e6f9bece7544bd549e2: Sending DeleteTablet for 3 replicas of tablet d2205cf858334d95aca2f2866ccce3ea
I20260928 08:46:07.503335 12147 tablet_service.cc:1514] Processing DeleteTablet for tablet d2205cf858334d95aca2f2866ccce3ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:46:07 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36774
I20260928 08:46:07.503372 12013 tablet_service.cc:1514] Processing DeleteTablet for tablet d2205cf858334d95aca2f2866ccce3ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:46:07 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33506
I20260928 08:46:07.503482 11870 tablet_service.cc:1514] Processing DeleteTablet for tablet d2205cf858334d95aca2f2866ccce3ea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:46:07 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41744
I20260928 08:46:07.506686 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 11802
I20260928 08:46:07.517006 12255 tablet_replica.cc:330] T d2205cf858334d95aca2f2866ccce3ea P d6f64b14e16e4b11bb7afdd7811dfc38: stopping tablet replica
I20260928 08:46:07.517670 12254 tablet_replica.cc:330] T d2205cf858334d95aca2f2866ccce3ea P 82e8686fb2d14bcaafe6c1d005f098bf: stopping tablet replica
I20260928 08:46:07.518666 12254 raft_consensus.cc:2229] T d2205cf858334d95aca2f2866ccce3ea P 82e8686fb2d14bcaafe6c1d005f098bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:46:07.519273 12255 raft_consensus.cc:2229] T d2205cf858334d95aca2f2866ccce3ea P d6f64b14e16e4b11bb7afdd7811dfc38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:46:07.520234 12254 raft_consensus.cc:2258] T d2205cf858334d95aca2f2866ccce3ea P 82e8686fb2d14bcaafe6c1d005f098bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:46:07.520557 12255 raft_consensus.cc:2258] T d2205cf858334d95aca2f2866ccce3ea P d6f64b14e16e4b11bb7afdd7811dfc38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:46:07.523434 12255 ts_tablet_manager.cc:1905] T d2205cf858334d95aca2f2866ccce3ea P d6f64b14e16e4b11bb7afdd7811dfc38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:46:07.524362 12256 tablet_replica.cc:330] T d2205cf858334d95aca2f2866ccce3ea P b2cc3f4599054772be08f0e661847282: stopping tablet replica
I20260928 08:46:07.524817 12256 raft_consensus.cc:2229] T d2205cf858334d95aca2f2866ccce3ea P b2cc3f4599054772be08f0e661847282 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:46:07.525661 12256 raft_consensus.cc:2258] T d2205cf858334d95aca2f2866ccce3ea P b2cc3f4599054772be08f0e661847282 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:46:07.526270 12254 ts_tablet_manager.cc:1905] T d2205cf858334d95aca2f2866ccce3ea P 82e8686fb2d14bcaafe6c1d005f098bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:46:07.527570 12256 ts_tablet_manager.cc:1905] T d2205cf858334d95aca2f2866ccce3ea P b2cc3f4599054772be08f0e661847282: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:46:07.530768 12255 ts_tablet_manager.cc:1918] T d2205cf858334d95aca2f2866ccce3ea P d6f64b14e16e4b11bb7afdd7811dfc38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:46:07.530927 12255 log.cc:1196] T d2205cf858334d95aca2f2866ccce3ea P d6f64b14e16e4b11bb7afdd7811dfc38: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-2/wal/wals/d2205cf858334d95aca2f2866ccce3ea
I20260928 08:46:07.531412 12255 ts_tablet_manager.cc:1939] T d2205cf858334d95aca2f2866ccce3ea P d6f64b14e16e4b11bb7afdd7811dfc38: Deleting consensus metadata
I20260928 08:46:07.533504 12254 ts_tablet_manager.cc:1918] T d2205cf858334d95aca2f2866ccce3ea P 82e8686fb2d14bcaafe6c1d005f098bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:46:07.533457 11709 catalog_manager.cc:4841] TS d6f64b14e16e4b11bb7afdd7811dfc38 (127.0.97.3:37457): tablet d2205cf858334d95aca2f2866ccce3ea (table test-table [id=c86aa0f0816d499abd326ad4148d4605]) successfully deleted
I20260928 08:46:07.533653 12254 log.cc:1196] T d2205cf858334d95aca2f2866ccce3ea P 82e8686fb2d14bcaafe6c1d005f098bf: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-1/wal/wals/d2205cf858334d95aca2f2866ccce3ea
Sep 28 08:46:07 dist-test-slave-n6xs krb5kdc[11246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:46:07.534111 12254 ts_tablet_manager.cc:1939] T d2205cf858334d95aca2f2866ccce3ea P 82e8686fb2d14bcaafe6c1d005f098bf: Deleting consensus metadata
I20260928 08:46:07.535174 12256 ts_tablet_manager.cc:1918] T d2205cf858334d95aca2f2866ccce3ea P b2cc3f4599054772be08f0e661847282: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:46:07.535357 12256 log.cc:1196] T d2205cf858334d95aca2f2866ccce3ea P b2cc3f4599054772be08f0e661847282: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790585022442112-388-0/minicluster-data/ts-0/wal/wals/d2205cf858334d95aca2f2866ccce3ea
I20260928 08:46:07.536000 12256 ts_tablet_manager.cc:1939] T d2205cf858334d95aca2f2866ccce3ea P b2cc3f4599054772be08f0e661847282: Deleting consensus metadata
I20260928 08:46:07.536125 11707 catalog_manager.cc:4841] TS 82e8686fb2d14bcaafe6c1d005f098bf (127.0.97.2:37281): tablet d2205cf858334d95aca2f2866ccce3ea (table test-table [id=c86aa0f0816d499abd326ad4148d4605]) successfully deleted
I20260928 08:46:07.537544 11707 catalog_manager.cc:4841] TS b2cc3f4599054772be08f0e661847282 (127.0.97.1:36871): tablet d2205cf858334d95aca2f2866ccce3ea (table test-table [id=c86aa0f0816d499abd326ad4148d4605]) successfully deleted
I20260928 08:46:07.542429 12226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:07.508383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41760 (local address 127.0.97.1:36871)
0928 08:46:07.508564 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 08:46:07.508571 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:46:07.522140 (+ 13569us) server_negotiation.cc:407] Connection header received
0928 08:46:07.522610 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:46:07.522617 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:46:07.522703 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:46:07.522778 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:46:07.524809 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:07.526006 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:07.528129 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:07.528829 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:07.528928 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:07.534922 (+ 5994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:46:07.534936 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:46:07.534954 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:46:07.535010 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:46:07.539568 (+ 4558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:46:07.540339 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:46:07.540351 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:46:07.540355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:46:07.540526 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:46:07.540903 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:46:07.540908 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:46:07.540911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:46:07.541349 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:46:07.541665 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0928 08:46:07.541824 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 08:46:07.542071 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 08:46:07.543196 11930 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:46:07.640084 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 11802
I20260928 08:46:07.661742 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 11941
Sep 28 08:46:07 dist-test-slave-n6xs krb5kdc[11246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 08:46:07.681360 12230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:07.662518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33510 (local address 127.0.97.2:37281)
0928 08:46:07.662691 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 08:46:07.662695 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:46:07.662715 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:46:07.662880 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:46:07.662886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:46:07.662976 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:46:07.663065 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:46:07.664622 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:07.665466 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:07.666817 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:07.667197 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:07.667293 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:07.673170 (+ 5877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:46:07.673188 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:46:07.673210 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:46:07.673255 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:46:07.677756 (+ 4501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:46:07.678596 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:46:07.678608 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:46:07.678611 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:46:07.679042 (+ 431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:46:07.679455 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:46:07.679460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:46:07.679463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:46:07.680226 (+ 763us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:46:07.680519 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0928 08:46:07.680917 (+ 398us) server_negotiation.cc:299] Negotiation successful
0928 08:46:07.681101 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 08:46:07.682590 12073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:46:07.784768 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 11941
I20260928 08:46:07.801930 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 12084
Sep 28 08:46:07 dist-test-slave-n6xs krb5kdc[11246](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 08:46:07.817181 12228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:07.802695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36784 (local address 127.0.97.3:37457)
0928 08:46:07.802879 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 08:46:07.802887 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:46:07.803019 (+ 132us) server_negotiation.cc:407] Connection header received
0928 08:46:07.803235 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:46:07.803239 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:46:07.803320 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:46:07.803427 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:46:07.804798 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:07.805534 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:07.806870 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:07.807252 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:07.807381 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:07.811743 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:46:07.811757 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:46:07.811773 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:46:07.811852 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:46:07.814350 (+ 2498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:46:07.814973 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:46:07.814989 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:46:07.814992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:46:07.815114 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:46:07.815475 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:46:07.815482 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:46:07.815486 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:46:07.816026 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:46:07.816321 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0928 08:46:07.816645 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 08:46:07.816859 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 08:46:07.817874 12207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:46:07.905682 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 12084
I20260928 08:46:07.922294 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 11690
I20260928 08:46:07.923503 11752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:46:08.033514 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 11690
I20260928 08:46:08.055958 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 08:46:08.390959 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 08:46:08.391251 388 mini_ranger.cc:66] Stopping Ranger...
I20260928 08:46:08.736608 388 mini_ranger.cc:68] Stopped Ranger
2026-09-28 08:46:08.736 UTC [11276] LOG: received smart shutdown request
2026-09-28 08:46:08.740 UTC [11276] LOG: background worker "logical replication launcher" (PID 11285) exited with exit code 1
2026-09-28 08:46:08.741 UTC [11280] LOG: shutting down
2026-09-28 08:46:08.912 UTC [11276] LOG: database system is shut down
2026-09-28T08:46:08Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49271 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:46:09 dist-test-slave-n6xs krb5kdc[12270](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:46:09 dist-test-slave-n6xs krb5kdc[12270](info): set up 2 sockets
Sep 28 08:46:09 dist-test-slave-n6xs krb5kdc[12270](info): commencing operation
krb5kdc: starting...
W20260928 08:46:10.184095 388 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:46:10 dist-test-slave-n6xs krb5kdc[12270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:46:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:46:10Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/rangeradmin_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/rangerlookup_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
I20260928 08:46:10.498624 388 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskUfDo5M/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/postgres -l logfile start
2026-09-28 08:46:13.105 UTC [12300] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 08:46:13.106 UTC [12300] LOG: listening on IPv4 address "127.0.97.20", port 50087
2026-09-28 08:46:13.109 UTC [12300] LOG: listening on Unix socket "/tmp/.s.PGSQL.50087"
2026-09-28 08:46:13.133 UTC [12303] LOG: database system was shut down at 2026-09-28 08:46:11 UTC
2026-09-28 08:46:13.136 UTC [12300] LOG: database system is ready to accept connections
I20260928 08:46:14.115960 388 mini_postgres.cc:96] Postgres bound to 50087
2026-09-28 08:46:14.133 UTC [12311] FATAL: database "slave" does not exist
127.0.97.20:50087 - accepting connections
I20260928 08:46:14.133844 388 mini_ranger.cc:161] Starting Ranger...
I20260928 08:46:14.160465 388 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 08:46:14.669582 388 mini_ranger.cc:87] Created ranger Postgres database
I20260928 08:46:14.669723 388 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ranger-admin
2026-09-28 08:46:14,958 [I] DB FLAVOR :POSTGRES
2026-09-28 08:46:14,959 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 08:46:14,959 [I] Checking connection..
2026-09-28 08:46:14,959 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 08:46:15,210 [I] Checking connection passed.
2026-09-28 08:46:15,210 [I] --------- Verifying version history table ---------
2026-09-28 08:46:15,210 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 08:46:15,475 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 08:46:15,475 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 08:46:15,714 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 08:46:15,714 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 08:46:15,714 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50087/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 08:46:15.979 UTC [12375] WARNING: there is no transaction in progress
2026-09-28 08:46:15,986 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 08:46:15,986 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 08:46:16,223 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 08:46:16,223 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 08:46:16,224 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 08:46:16,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 08:46:16,720 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-n6xs.c.gcp-upstream.internal','N') ;"
2026-09-28 08:46:16,943 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 08:46:17,175 [I] Table x_portal_user does not exist in database ranger
2026-09-28 08:46:17,175 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 08:46:17,414 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 08:46:17,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 08:46:17,675 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 08:46:17,951 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 08:46:17,951 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50087/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 08:46:18.933 UTC [12483] WARNING: there is no transaction in progress
2026-09-28 08:46:18.951 UTC [12483] WARNING: there is no transaction in progress
2026-09-28 08:46:18.970 UTC [12483] WARNING: there is no transaction in progress
2026-09-28 08:46:18.987 UTC [12483] WARNING: there is no transaction in progress
2026-09-28 08:46:19.006 UTC [12483] WARNING: there is no transaction in progress
2026-09-28 08:46:19.029 UTC [12483] WARNING: there is no transaction in progress
2026-09-28 08:46:19.101 UTC [12483] WARNING: there is no transaction in progress
2026-09-28 08:46:19.108 UTC [12483] WARNING: there is no transaction in progress
2026-09-28 08:46:19.124 UTC [12483] WARNING: there is no transaction in progress
2026-09-28 08:46:19.134 UTC [12483] WARNING: there is no transaction in progress
2026-09-28 08:46:19.146 UTC [12483] WARNING: there is no transaction in progress
2026-09-28 08:46:19.162 UTC [12483] WARNING: there is no transaction in progress
2026-09-28 08:46:19.173 UTC [12483] WARNING: there is no transaction in progress
2026-09-28 08:46:19.180 UTC [12483] WARNING: there is no transaction in progress
2026-09-28 08:46:19.188 UTC [12483] WARNING: there is no transaction in progress
2026-09-28 08:46:19.195 UTC [12483] WARNING: there is no transaction in progress
2026-09-28 08:46:19,848 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 08:46:19,849 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 08:46:20,106 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 08:46:20,106 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 08:46:20,355 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 08:46:20,355 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 08:46:20,598 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 08:46:20,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 08:46:20,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 08:46:21,096 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-n6xs.c.gcp-upstream.internal';"
2026-09-28 08:46:21,352 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 08:46:21,352 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50087/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 08:46:21,597 [I] DB_PATCHES have already been applied
I20260928 08:46:21.603157 388 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ranger-admin:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskUfDo5M/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 08:46:21.603266 388 mini_ranger.cc:193] Using host: 127.0.97.20
I20260928 08:46:21.617660 388 mini_ranger.cc:227] Ranger admin URL: http://127.0.97.20:39981
Sep 28, 2026 8:46:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 8:46:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 8:46:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 8:46:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 8:46:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 8:46:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 8:46:22 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 08:46:22.769366 388 mini_ranger.cc:160] Time spent starting Ranger: real 8.636s user 0.005s sys 0.033s
Sep 28, 2026 8:46:42 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 8:46:42 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 8:46:42 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 8:46:42 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 8:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 8:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 8:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 8:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 8:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 8:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 8:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 8:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 8:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 8:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 8:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 8:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 8:46:42 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 8:46:43 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 08:46:44.403594 388 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.97.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.97.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.97.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/rangerkms_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.97.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.97.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.97.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/HTTP_127.0.97.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 08:46:44 dist-test-slave-n6xs krb5kdc[12270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585204, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 08:46:44.580271 388 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 08:46:44.603302 388 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 08:46:45.545964 388 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 08:46:45.546090 388 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ranger-kms
I20260928 08:46:45.546190 388 mini_ranger_kms.cc:227] Using postgres at 127.0.97.20:50087
2026-09-28 08:46:45,894 [I] DB FLAVOR :POSTGRES
2026-09-28 08:46:45,895 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 08:46:45,895 [I] Checking connection
2026-09-28 08:46:45,895 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50087/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 08:46:46,150 [I] connection success
2026-09-28 08:46:46,150 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 08:46:46,150 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 08:46:46,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50087/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 08:46:46,394 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 08:46:46,394 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 08:46:46,394 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 08:46:46,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.97.20:50087/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 08:46:46,661 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 08:46:47.107496 388 mini_ranger_kms.cc:310] Created kms service
I20260928 08:46:47.279309 388 mini_ranger_kms.cc:326] Created kudu user
I20260928 08:46:47.326340 388 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 08:46:47 dist-test-slave-n6xs krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585204, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 08:46:47.717130 388 mini_ranger_kms.cc:384] Added ranger policy
I20260928 08:46:47.717468 388 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ranger-kms:/tmp/dist-test-taskUfDo5M/build/asan/bin/postgresql.jar:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskUfDo5M/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 08:46:47.717536 388 mini_ranger_kms.cc:240] Using host: 127.0.97.20
I20260928 08:46:47.731984 388 mini_ranger_kms.cc:276] Ranger KMS PID: 12663
I20260928 08:46:47.732126 388 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.97.20:38227
Sep 28, 2026 8:46:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 8:46:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 8:46:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 8:46:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 8:46:49 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 8:46:49 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 08:46:50.135885 388 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.556s user 0.091s sys 0.272s
I20260928 08:46:50.136150 388 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 08:46:50.136317 388 mini_ranger_kms.cc:401] 127.0.97.20:38227/kms/v1/keys
Sep 28 08:46:50 dist-test-slave-n6xs krb5kdc[12270](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790585210, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 08:46:50 dist-test-slave-n6xs krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585210, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 28 08:46:51 dist-test-slave-n6xs krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585210, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.97.20@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Sep 28 08:46:53 dist-test-slave-n6xs krb5kdc[12270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:46:53.500407 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34293
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34293
--ranger_config_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:38227/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:46:53.828615 12714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:46:53.828895 12714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:46:53.829048 12714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:46:53.836220 12714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:46:53.836323 12714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:46:53.836404 12714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:46:53.836508 12714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:46:53.836588 12714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:46:53.847958 12714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.97.20:38227/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34293
--ranger_config_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:46:53.849321 12714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:46:53.850903 12714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:46:53.863890 12719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:46:53.863929 12722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:46:53.863863 12720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:46:53.864180 12714 server_base.cc:1034] running on GCE node
I20260928 08:46:53.865453 12714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:46:53.867038 12714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:46:53.868280 12714 hybrid_clock.cc:648] HybridClock initialized: now 1790585213868221 us; error 36 us; skew 500 ppm
I20260928 08:46:53.868625 12714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:46:53 dist-test-slave-n6xs krb5kdc[12270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:46:53.877619 12714 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:46:53.879237 12714 webserver.cc:460] Webserver started at http://127.0.97.62:44273/ using document root <none> and password file <none>
I20260928 08:46:53.879982 12714 fs_manager.cc:359] Metadata directory not provided
I20260928 08:46:53.880120 12714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:46:53.880421 12714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 08:46:53 dist-test-slave-n6xs krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585213, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 08:46:53.947427 12714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "2e6f3e4dd306497581f87a1072af880a"
format_stamp: "Formatted at 2026-09-28 08:46:53 on dist-test-slave-n6xs"
server_key: "d5f3ef0378fd9b10acb8947cf3414dbc"
server_key_iv: "a694fae9f72c94060fd5d1c6202f7d1f"
server_key_version: "kuduclusterkey@0"
I20260928 08:46:53.948344 12714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "2e6f3e4dd306497581f87a1072af880a"
format_stamp: "Formatted at 2026-09-28 08:46:53 on dist-test-slave-n6xs"
server_key: "d5f3ef0378fd9b10acb8947cf3414dbc"
server_key_iv: "a694fae9f72c94060fd5d1c6202f7d1f"
server_key_version: "kuduclusterkey@0"
I20260928 08:46:53.954020 12714 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:46:53.973981 12729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:46:53.975975 12714 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 08:46:53.976178 12714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "2e6f3e4dd306497581f87a1072af880a"
format_stamp: "Formatted at 2026-09-28 08:46:53 on dist-test-slave-n6xs"
server_key: "d5f3ef0378fd9b10acb8947cf3414dbc"
server_key_iv: "a694fae9f72c94060fd5d1c6202f7d1f"
server_key_version: "kuduclusterkey@0"
I20260928 08:46:53.976456 12714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:46:53.999928 12714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:46:54.004841 12714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:46:54.005199 12714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:46:54.030627 12714 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34293
I20260928 08:46:54.030632 12780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34293 every 8 connection(s)
I20260928 08:46:54.032361 12714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:46:54.034507 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 12714
I20260928 08:46:54.034770 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:46:54.035475 388 external_mini_cluster.cc:1428] Setting key ffd9c52952d7b13a8692be56d96b6796
I20260928 08:46:54.037223 12781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:46:54.052424 12781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a: Bootstrap starting.
I20260928 08:46:54.056631 12781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a: Neither blocks nor log segments found. Creating new log.
I20260928 08:46:54.058041 12781 log.cc:824] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a: Log is configured to *not* fsync() on all Append() calls
I20260928 08:46:54.061718 12781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a: No bootstrap required, opened a new log
Sep 28 08:46:54 dist-test-slave-n6xs krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:46:54.072532 12784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:54.038024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53162 (local address 127.0.97.62:34293)
0928 08:46:54.050484 (+ 12460us) server_negotiation.cc:206] Beginning negotiation
0928 08:46:54.050539 (+ 55us) server_negotiation.cc:399] Waiting for connection header
0928 08:46:54.051241 (+ 702us) server_negotiation.cc:407] Connection header received
0928 08:46:54.053489 (+ 2248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:46:54.053514 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:46:54.053961 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:46:54.054422 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:46:54.055985 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:54.057976 (+ 1991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:54.060077 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:54.060758 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:54.060965 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:54.064933 (+ 3968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:46:54.064943 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:46:54.064959 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:46:54.065001 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:46:54.068219 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:46:54.068930 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:46:54.068936 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:46:54.068943 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:46:54.069089 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:46:54.069513 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:46:54.069517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:46:54.069519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:46:54.070087 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:46:54.070431 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0928 08:46:54.070927 (+ 496us) server_negotiation.cc:299] Negotiation successful
0928 08:46:54.071404 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11702,"thread_start_us":11853,"threads_started":1}
I20260928 08:46:54.072896 12781 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6f3e4dd306497581f87a1072af880a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34293 } }
I20260928 08:46:54.073388 12781 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:46:54.073616 12781 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e6f3e4dd306497581f87a1072af880a, State: Initialized, Role: FOLLOWER
I20260928 08:46:54.074365 12781 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6f3e4dd306497581f87a1072af880a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34293 } }
I20260928 08:46:54.074666 12781 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:46:54.074831 12781 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:46:54.075021 12781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:46:54.078282 12781 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6f3e4dd306497581f87a1072af880a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34293 } }
I20260928 08:46:54.078862 12781 leader_election.cc:302] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e6f3e4dd306497581f87a1072af880a; no voters:
I20260928 08:46:54.079550 12781 leader_election.cc:288] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:46:54.079634 12786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:46:54.080681 12786 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a [term 1 LEADER]: Becoming Leader. State: Replica: 2e6f3e4dd306497581f87a1072af880a, State: Running, Role: LEADER
I20260928 08:46:54.081254 12786 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6f3e4dd306497581f87a1072af880a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34293 } }
I20260928 08:46:54.081905 12781 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:46:54.084506 12787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e6f3e4dd306497581f87a1072af880a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6f3e4dd306497581f87a1072af880a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34293 } } }
I20260928 08:46:54.084893 12787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a [sys.catalog]: This master's current role is: LEADER
I20260928 08:46:54.085381 12788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e6f3e4dd306497581f87a1072af880a. Latest consensus state: current_term: 1 leader_uuid: "2e6f3e4dd306497581f87a1072af880a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6f3e4dd306497581f87a1072af880a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34293 } } }
I20260928 08:46:54.085664 12788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a [sys.catalog]: This master's current role is: LEADER
I20260928 08:46:54.092525 12791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:46:54.101322 12791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:46:54.126250 12791 catalog_manager.cc:1347] Generated new cluster ID: 51e53e40f9da451cba3ed97ef493dcb0
I20260928 08:46:54.126392 12791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:46:54.129701 12781 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 08:46:54.170640 12791 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:46:54.172817 12791 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:46:54.187907 12791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a: Generated new TSK 0
I20260928 08:46:54.188995 12791 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:46:54.931063 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:38227/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34293
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:46:55 dist-test-slave-n6xs krb5kdc[12270](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790585215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 08:46:55.340370 12826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:46:55.340773 12826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:46:55.340983 12826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:46:55.352631 12826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:46:55.352775 12826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:46:55.353012 12826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:46:55.368497 12826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.97.20:38227/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:46:55.370397 12826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:46:55.372169 12826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:46:55.391976 12837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:46:55.396178 12836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:46:55.414440 12826 server_base.cc:1034] running on GCE node
W20260928 08:46:55.415375 12839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:46:55.416777 12826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:46:55.422740 12826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:46:55.424044 12826 hybrid_clock.cc:648] HybridClock initialized: now 1790585215423986 us; error 54 us; skew 500 ppm
I20260928 08:46:55.424464 12826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:46:55 dist-test-slave-n6xs krb5kdc[12270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:46:55.428886 12826 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:46:55.431010 12826 webserver.cc:460] Webserver started at http://127.0.97.1:40335/ using document root <none> and password file <none>
I20260928 08:46:55.432034 12826 fs_manager.cc:359] Metadata directory not provided
I20260928 08:46:55.432179 12826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:46:55.432600 12826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 08:46:55 dist-test-slave-n6xs krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 08:46:55.452984 12826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-0/data/instance:
uuid: "5acbab1c85d348bca5d92cee3682e166"
format_stamp: "Formatted at 2026-09-28 08:46:55 on dist-test-slave-n6xs"
server_key: "f8ef3427642d4caf506742e24e7932e8"
server_key_iv: "9fa2b721c2cb54c13fb57c6285d10e3b"
server_key_version: "kuduclusterkey@0"
I20260928 08:46:55.454149 12826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-0/wal/instance:
uuid: "5acbab1c85d348bca5d92cee3682e166"
format_stamp: "Formatted at 2026-09-28 08:46:55 on dist-test-slave-n6xs"
server_key: "f8ef3427642d4caf506742e24e7932e8"
server_key_iv: "9fa2b721c2cb54c13fb57c6285d10e3b"
server_key_version: "kuduclusterkey@0"
I20260928 08:46:55.460937 12826 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260928 08:46:55.492048 12846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:46:55.493992 12826 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.002s sys 0.006s
I20260928 08:46:55.494205 12826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-0/wal
uuid: "5acbab1c85d348bca5d92cee3682e166"
format_stamp: "Formatted at 2026-09-28 08:46:55 on dist-test-slave-n6xs"
server_key: "f8ef3427642d4caf506742e24e7932e8"
server_key_iv: "9fa2b721c2cb54c13fb57c6285d10e3b"
server_key_version: "kuduclusterkey@0"
I20260928 08:46:55.494498 12826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:46:55.509971 12826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:46:55.514384 12826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:46:55.514689 12826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:46:55.515733 12826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:46:55.523582 12826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:46:55.523694 12826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:46:55.523847 12826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:46:55.523914 12826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260928 08:46:55.598101 12826 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38533
I20260928 08:46:55.599138 12958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38533 every 8 connection(s)
I20260928 08:46:55.599826 12826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:46:55.605801 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 12826
I20260928 08:46:55.606047 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:46:55.607128 388 external_mini_cluster.cc:1428] Setting key d2c51e0d4e0766857a4d68c8645318c2
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Sep 28 08:46:55 dist-test-slave-n6xs krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:46:55.665493 12962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:55.604995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34293 (local address 127.0.97.1:48265)
0928 08:46:55.617976 (+ 12981us) negotiation.cc:107] Waiting for socket to connect
0928 08:46:55.618036 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 08:46:55.631555 (+ 13519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:46:55.632782 (+ 1227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:46:55.632794 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:46:55.633486 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:46:55.634334 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:46:55.634358 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:46:55.648413 (+ 14055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:46:55.648419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:46:55.650107 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:46:55.650140 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:46:55.651781 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:46:55.651792 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:46:55.652198 (+ 406us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:55.653098 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:46:55.653134 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:46:55.656148 (+ 3014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:46:55.660358 (+ 4210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:46:55.660370 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:46:55.660392 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:46:55.661053 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:46:55.661536 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:46:55.661544 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:46:55.661551 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:46:55.661846 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:46:55.662588 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:46:55.662599 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:46:55.663093 (+ 494us) client_negotiation.cc:769] Sending connection context
0928 08:46:55.663405 (+ 312us) client_negotiation.cc:240] Negotiation successful
0928 08:46:55.663986 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12605,"thread_start_us":14757,"threads_started":1}
I20260928 08:46:55.666923 12961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:55.604406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:48265 (local address 127.0.97.62:34293)
0928 08:46:55.631374 (+ 26968us) server_negotiation.cc:206] Beginning negotiation
0928 08:46:55.631381 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:46:55.631411 (+ 30us) server_negotiation.cc:407] Connection header received
0928 08:46:55.632092 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:46:55.632098 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:46:55.632218 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:46:55.632321 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:46:55.634532 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:55.648155 (+ 13623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:55.650310 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:55.650987 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:55.651122 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:55.656441 (+ 5319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:46:55.656448 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:46:55.656453 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:46:55.656496 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:46:55.660183 (+ 3687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:46:55.661217 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:46:55.661226 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:46:55.661235 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:46:55.661402 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:46:55.661981 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:46:55.661989 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:46:55.661995 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:46:55.662360 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:46:55.662515 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 08:46:55.666216 (+ 3701us) server_negotiation.cc:299] Negotiation successful
0928 08:46:55.666479 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26827,"thread_start_us":26683,"threads_started":1}
I20260928 08:46:55.669397 12959 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34293
I20260928 08:46:55.669723 12959 heartbeater.cc:464] Registering TS with master...
I20260928 08:46:55.670619 12959 heartbeater.cc:511] Master 127.0.97.62:34293 requested a full tablet report, sending...
I20260928 08:46:55.673249 12746 ts_manager.cc:194] Registered new tserver with Master: 5acbab1c85d348bca5d92cee3682e166 (127.0.97.1:38533)
I20260928 08:46:55.675513 12746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:48265
Sep 28 08:46:55 dist-test-slave-n6xs krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 08:46:55.709121 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:38227/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34293
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:46:56 dist-test-slave-n6xs krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
W20260928 08:46:56.206365 12965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:46:56.206876 12965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:46:56.207077 12965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:46:56.218720 12965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:46:56.218932 12965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:46:56.219158 12965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:46:56.234319 12965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.97.20:38227/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:46:56.236373 12965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:46:56.238116 12965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:46:56.277201 12972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:46:56.278801 12973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:46:56.286270 12976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:46:56.286759 12965 server_base.cc:1034] running on GCE node
I20260928 08:46:56.287698 12965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:46:56.292457 12965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:46:56.293723 12965 hybrid_clock.cc:648] HybridClock initialized: now 1790585216293625 us; error 93 us; skew 500 ppm
I20260928 08:46:56.294134 12965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:46:56 dist-test-slave-n6xs krb5kdc[12270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:46:56.298367 12965 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:46:56.321427 12965 webserver.cc:460] Webserver started at http://127.0.97.2:45017/ using document root <none> and password file <none>
I20260928 08:46:56.322467 12965 fs_manager.cc:359] Metadata directory not provided
I20260928 08:46:56.322665 12965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:46:56.323021 12965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 08:46:56 dist-test-slave-n6xs krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 08:46:56.352610 12965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-1/data/instance:
uuid: "24be679363434337b186be2bbded83e9"
format_stamp: "Formatted at 2026-09-28 08:46:56 on dist-test-slave-n6xs"
server_key: "ee2c6263b1eee0f59699988e786a392e"
server_key_iv: "3c2b8240a4e3db2d98950b3ed2598224"
server_key_version: "kuduclusterkey@0"
I20260928 08:46:56.353745 12965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-1/wal/instance:
uuid: "24be679363434337b186be2bbded83e9"
format_stamp: "Formatted at 2026-09-28 08:46:56 on dist-test-slave-n6xs"
server_key: "ee2c6263b1eee0f59699988e786a392e"
server_key_iv: "3c2b8240a4e3db2d98950b3ed2598224"
server_key_version: "kuduclusterkey@0"
I20260928 08:46:56.367106 12965 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.003s sys 0.008s
I20260928 08:46:56.396870 12989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:46:56.398919 12965 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.008s sys 0.000s
I20260928 08:46:56.399236 12965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-1/wal
uuid: "24be679363434337b186be2bbded83e9"
format_stamp: "Formatted at 2026-09-28 08:46:56 on dist-test-slave-n6xs"
server_key: "ee2c6263b1eee0f59699988e786a392e"
server_key_iv: "3c2b8240a4e3db2d98950b3ed2598224"
server_key_version: "kuduclusterkey@0"
I20260928 08:46:56.399593 12965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:46:56.442028 12965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:46:56.447320 12965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:46:56.447731 12965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:46:56.455179 12965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:46:56.463346 12965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:46:56.463519 12965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:46:56.463696 12965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:46:56.463846 12965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:46:56.527338 12965 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36277
I20260928 08:46:56.527369 13101 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36277 every 8 connection(s)
I20260928 08:46:56.528821 12965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:46:56.533083 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 12965
I20260928 08:46:56.533274 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:46:56.534026 388 external_mini_cluster.cc:1428] Setting key c40648499bc4cadfbcb3b2a452401304
Sep 28 08:46:56 dist-test-slave-n6xs krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260928 08:46:56.580246 13105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:56.537336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55873 (local address 127.0.97.62:34293)
0928 08:46:56.558293 (+ 20957us) server_negotiation.cc:206] Beginning negotiation
0928 08:46:56.558301 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:46:56.558329 (+ 28us) server_negotiation.cc:407] Connection header received
0928 08:46:56.560349 (+ 2020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:46:56.560358 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:46:56.560474 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:46:56.560559 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:46:56.563003 (+ 2444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:56.564270 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:56.567042 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:56.567786 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:56.567950 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:56.572629 (+ 4679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:46:56.572639 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:46:56.572647 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:46:56.572694 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:46:56.576094 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:46:56.577090 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:46:56.577098 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:46:56.577104 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:46:56.577257 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:46:56.577803 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:46:56.577811 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:46:56.577816 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:46:56.578206 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:46:56.578331 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 08:46:56.579401 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0928 08:46:56.579680 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20821,"thread_start_us":20995,"threads_started":1}
I20260928 08:46:56.582780 13104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:56.532920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34293 (local address 127.0.97.2:55873)
0928 08:46:56.556814 (+ 23894us) negotiation.cc:107] Waiting for socket to connect
0928 08:46:56.556873 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 08:46:56.560009 (+ 3136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:46:56.561136 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:46:56.561151 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:46:56.561868 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:46:56.562782 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:46:56.562809 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:46:56.564501 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:46:56.564509 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:46:56.566860 (+ 2351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:46:56.566876 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:46:56.568061 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:46:56.568068 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:46:56.568392 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:56.569286 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:46:56.569325 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:46:56.572342 (+ 3017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:46:56.576263 (+ 3921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:46:56.576275 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:46:56.576297 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:46:56.576949 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:46:56.577396 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:46:56.577403 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:46:56.577409 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:46:56.577674 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:46:56.578449 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:46:56.578460 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:46:56.578947 (+ 487us) client_negotiation.cc:769] Sending connection context
0928 08:46:56.580520 (+ 1573us) client_negotiation.cc:240] Negotiation successful
0928 08:46:56.581101 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23559,"thread_start_us":23298,"threads_started":1}
I20260928 08:46:56.584828 13102 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34293
I20260928 08:46:56.585178 13102 heartbeater.cc:464] Registering TS with master...
I20260928 08:46:56.585881 13102 heartbeater.cc:511] Master 127.0.97.62:34293 requested a full tablet report, sending...
I20260928 08:46:56.587461 12746 ts_manager.cc:194] Registered new tserver with Master: 24be679363434337b186be2bbded83e9 (127.0.97.2:36277)
I20260928 08:46:56.588793 12746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:46:56.607921 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:38227/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34293
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:46:56.679738 12959 heartbeater.cc:503] Master 127.0.97.62:34293 was elected leader, sending a full tablet report...
W20260928 08:46:56.972421 13108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:46:56.972813 13108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:46:56.972995 13108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:46:56.980086 13108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:46:56.980247 13108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:46:56.980432 13108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:46:56.990386 13108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.97.20:38227/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:34293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:46:56.991683 13108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:46:56.993002 13108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:46:57.005872 13114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:46:57.005931 13113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:46:57.005820 13116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:46:57.007045 13108 server_base.cc:1034] running on GCE node
I20260928 08:46:57.007664 13108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:46:57.008581 13108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:46:57.009773 13108 hybrid_clock.cc:648] HybridClock initialized: now 1790585217009731 us; error 30 us; skew 500 ppm
I20260928 08:46:57.010077 13108 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:46:57 dist-test-slave-n6xs krb5kdc[12270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:46:57.013182 13108 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:46:57.014596 13108 webserver.cc:460] Webserver started at http://127.0.97.3:34379/ using document root <none> and password file <none>
I20260928 08:46:57.015331 13108 fs_manager.cc:359] Metadata directory not provided
I20260928 08:46:57.015427 13108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:46:57.015661 13108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 08:46:57 dist-test-slave-n6xs krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 08:46:57.030478 13108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-2/data/instance:
uuid: "d6b545ca970645febef66643334ce4ae"
format_stamp: "Formatted at 2026-09-28 08:46:57 on dist-test-slave-n6xs"
server_key: "00c8d55f827f4f097658856ccc346c63"
server_key_iv: "f1d13f52921a5c3b44664ca7db58c130"
server_key_version: "kuduclusterkey@0"
I20260928 08:46:57.031553 13108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-2/wal/instance:
uuid: "d6b545ca970645febef66643334ce4ae"
format_stamp: "Formatted at 2026-09-28 08:46:57 on dist-test-slave-n6xs"
server_key: "00c8d55f827f4f097658856ccc346c63"
server_key_iv: "f1d13f52921a5c3b44664ca7db58c130"
server_key_version: "kuduclusterkey@0"
I20260928 08:46:57.038072 13108 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.002s
I20260928 08:46:57.054935 13123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:46:57.056710 13108 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:46:57.056922 13108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-2/wal
uuid: "d6b545ca970645febef66643334ce4ae"
format_stamp: "Formatted at 2026-09-28 08:46:57 on dist-test-slave-n6xs"
server_key: "00c8d55f827f4f097658856ccc346c63"
server_key_iv: "f1d13f52921a5c3b44664ca7db58c130"
server_key_version: "kuduclusterkey@0"
I20260928 08:46:57.057181 13108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:46:57.073378 13108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:46:57.077371 13108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:46:57.077710 13108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:46:57.078723 13108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:46:57.080112 13108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:46:57.080212 13108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:46:57.080355 13108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:46:57.080425 13108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:46:57.114095 13108 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38361
I20260928 08:46:57.114315 13235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38361 every 8 connection(s)
I20260928 08:46:57.115414 13108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:46:57.116858 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 13108
I20260928 08:46:57.117039 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:46:57.117518 388 external_mini_cluster.cc:1428] Setting key 2ae2ff75a85565235c72af46e61e4649
Sep 28 08:46:57 dist-test-slave-n6xs krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:46:57.135970 13238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:57.118046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35739 (local address 127.0.97.62:34293)
0928 08:46:57.118620 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0928 08:46:57.118637 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 08:46:57.119664 (+ 1027us) server_negotiation.cc:407] Connection header received
0928 08:46:57.121498 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:46:57.121502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:46:57.121565 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:46:57.121662 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:46:57.123071 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:57.123876 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:57.125522 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:57.125956 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:57.126066 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:57.129352 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:46:57.129356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:46:57.129359 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:46:57.129392 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:46:57.133004 (+ 3612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:46:57.133854 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:46:57.133858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:46:57.133860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:46:57.133994 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:46:57.134351 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:46:57.134355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:46:57.134357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:46:57.134686 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:46:57.134812 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 08:46:57.135396 (+ 584us) server_negotiation.cc:299] Negotiation successful
0928 08:46:57.135580 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":294,"threads_started":1}
I20260928 08:46:57.136855 13239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:57.118621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34293 (local address 127.0.97.3:35739)
0928 08:46:57.119489 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0928 08:46:57.119542 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 08:46:57.121259 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:46:57.121899 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:46:57.121913 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:46:57.122327 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:46:57.122890 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:46:57.122907 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:46:57.124027 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:46:57.124031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:46:57.125394 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:46:57.125405 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:46:57.126156 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:46:57.126161 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:46:57.126344 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:57.127003 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:46:57.127024 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:46:57.129133 (+ 2109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:46:57.133127 (+ 3994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:46:57.133133 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:46:57.133146 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:46:57.133748 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:46:57.134091 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:46:57.134094 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:46:57.134096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:46:57.134261 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:46:57.134798 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:46:57.134804 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:46:57.135079 (+ 275us) client_negotiation.cc:769] Sending connection context
0928 08:46:57.135302 (+ 223us) client_negotiation.cc:240] Negotiation successful
0928 08:46:57.135664 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":382,"threads_started":1}
I20260928 08:46:57.138309 13236 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34293
I20260928 08:46:57.138607 13236 heartbeater.cc:464] Registering TS with master...
I20260928 08:46:57.139307 13236 heartbeater.cc:511] Master 127.0.97.62:34293 requested a full tablet report, sending...
I20260928 08:46:57.140714 12746 ts_manager.cc:194] Registered new tserver with Master: d6b545ca970645febef66643334ce4ae (127.0.97.3:38361)
I20260928 08:46:57.141494 12746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:35739
I20260928 08:46:57.145699 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:46:57.147312 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 12826
Sep 28 08:46:57 dist-test-slave-n6xs krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:46:57.163553 13241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:57.148115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58154 (local address 127.0.97.1:38533)
0928 08:46:57.148754 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0928 08:46:57.148761 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:46:57.148785 (+ 24us) server_negotiation.cc:407] Connection header received
0928 08:46:57.148943 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:46:57.148950 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:46:57.149147 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:46:57.149273 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:46:57.150770 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:57.151778 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:57.153251 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:57.153777 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:57.153872 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:57.157828 (+ 3956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:46:57.157837 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:46:57.157850 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:46:57.157880 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:46:57.160717 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:46:57.161396 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:46:57.161402 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:46:57.161404 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:46:57.161579 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:46:57.162025 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:46:57.162030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:46:57.162040 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:46:57.162444 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:46:57.162718 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0928 08:46:57.162970 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 08:46:57.163160 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":301,"threads_started":1}
I20260928 08:46:57.164829 12954 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:46:57.248631 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 12826
I20260928 08:46:57.265343 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 12965
Sep 28 08:46:57 dist-test-slave-n6xs krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 08:46:57.281719 13243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:57.266200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51926 (local address 127.0.97.2:36277)
0928 08:46:57.266677 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0928 08:46:57.266685 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:46:57.266709 (+ 24us) server_negotiation.cc:407] Connection header received
0928 08:46:57.266767 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:46:57.266772 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:46:57.266977 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:46:57.267174 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:46:57.268452 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:57.269417 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:57.270751 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:57.271291 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:57.271428 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:57.275591 (+ 4163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:46:57.275606 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:46:57.275621 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:46:57.275664 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:46:57.278829 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:46:57.279574 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:46:57.279582 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:46:57.279584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:46:57.279754 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:46:57.280185 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:46:57.280189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:46:57.280191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:46:57.280531 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:46:57.280763 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 08:46:57.281118 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 08:46:57.281379 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":222,"threads_started":1}
I20260928 08:46:57.283280 13097 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:46:57.365147 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 12965
I20260928 08:46:57.380872 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 13108
Sep 28 08:46:57 dist-test-slave-n6xs krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585213, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 08:46:57.396890 13245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:57.381917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34242 (local address 127.0.97.3:38361)
0928 08:46:57.382594 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0928 08:46:57.382609 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 08:46:57.382683 (+ 74us) server_negotiation.cc:407] Connection header received
0928 08:46:57.382780 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:46:57.382790 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:46:57.383029 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:46:57.383160 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:46:57.384576 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:57.385558 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:57.386775 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:57.387294 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:57.387388 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:57.391587 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:46:57.391598 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:46:57.391611 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:46:57.391679 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:46:57.394299 (+ 2620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:46:57.394937 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:46:57.394944 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:46:57.394946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:46:57.395059 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:46:57.395358 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:46:57.395361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:46:57.395363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:46:57.395681 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:46:57.395993 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0928 08:46:57.396260 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 08:46:57.396476 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":344,"threads_started":1}
I20260928 08:46:57.398181 13231 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:46:57.483024 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 13108
I20260928 08:46:57.498548 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 12714
I20260928 08:46:57.500025 12776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:46:57.607666 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 12714
I20260928 08:46:57.629311 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34293
--webserver_interface=127.0.97.62
--webserver_port=44273
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34293
--ranger_config_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:38227/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:38227/kms with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:46:57.952481 13248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:46:57.952764 13248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:46:57.952852 13248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:46:57.964932 13248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:46:57.965147 13248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:46:57.965291 13248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:46:57.965404 13248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:46:57.965515 13248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:46:57.977547 13248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:38227/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34293
--ranger_config_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=44273
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:46:57.979395 13248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:46:57.981096 13248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:46:57.998471 13253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:46:58.000063 13254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:46:58.000869 13256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:46:58.001603 13248 server_base.cc:1034] running on GCE node
I20260928 08:46:58.002557 13248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:46:58.008101 13248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:46:58.009428 13248 hybrid_clock.cc:648] HybridClock initialized: now 1790585218009361 us; error 61 us; skew 500 ppm
I20260928 08:46:58.010025 13248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:46:58 dist-test-slave-n6xs krb5kdc[12270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:46:58.014761 13248 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:46:58.016759 13248 webserver.cc:460] Webserver started at http://127.0.97.62:44273/ using document root <none> and password file <none>
I20260928 08:46:58.017685 13248 fs_manager.cc:359] Metadata directory not provided
I20260928 08:46:58.017822 13248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 08:46:58 dist-test-slave-n6xs krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 08:46:58.087750 13248 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 08:46:58.091022 13264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:46:58.092664 13248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:46:58.092882 13248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "2e6f3e4dd306497581f87a1072af880a"
format_stamp: "Formatted at 2026-09-28 08:46:53 on dist-test-slave-n6xs"
server_key: "d5f3ef0378fd9b10acb8947cf3414dbc"
server_key_iv: "a694fae9f72c94060fd5d1c6202f7d1f"
server_key_version: "kuduclusterkey@0"
I20260928 08:46:58.093988 13248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:46:58.125692 13248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:46:58.129953 13248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:46:58.130254 13248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:46:58.149302 13248 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34293
I20260928 08:46:58.149336 13315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34293 every 8 connection(s)
I20260928 08:46:58.150928 13248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:46:58.152241 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 13248
I20260928 08:46:58.153329 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:38227/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:38533
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:34293
--webserver_port=40335
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:38227/kms with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:46:58.160642 13316 sys_catalog.cc:263] Verifying existing consensus state
I20260928 08:46:58.163856 13316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a: Bootstrap starting.
I20260928 08:46:58.182391 13316 log.cc:824] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a: Log is configured to *not* fsync() on all Append() calls
I20260928 08:46:58.191095 13316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:46:58.191612 13316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a: Bootstrap complete.
I20260928 08:46:58.200909 13316 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6f3e4dd306497581f87a1072af880a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34293 } }
I20260928 08:46:58.201725 13316 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e6f3e4dd306497581f87a1072af880a, State: Initialized, Role: FOLLOWER
I20260928 08:46:58.202370 13316 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6f3e4dd306497581f87a1072af880a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34293 } }
I20260928 08:46:58.202589 13316 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:46:58.202711 13316 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:46:58.202893 13316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a [term 1 FOLLOWER]: Advancing to term 2
I20260928 08:46:58.206568 13316 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6f3e4dd306497581f87a1072af880a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34293 } }
I20260928 08:46:58.207041 13316 leader_election.cc:302] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e6f3e4dd306497581f87a1072af880a; no voters:
I20260928 08:46:58.207656 13316 leader_election.cc:288] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 08:46:58.207831 13320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a [term 2 FOLLOWER]: Leader election won for term 2
I20260928 08:46:58.208863 13320 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a [term 2 LEADER]: Becoming Leader. State: Replica: 2e6f3e4dd306497581f87a1072af880a, State: Running, Role: LEADER
I20260928 08:46:58.209460 13320 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6f3e4dd306497581f87a1072af880a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34293 } }
I20260928 08:46:58.209852 13316 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:46:58.211938 13322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e6f3e4dd306497581f87a1072af880a. Latest consensus state: current_term: 2 leader_uuid: "2e6f3e4dd306497581f87a1072af880a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6f3e4dd306497581f87a1072af880a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34293 } } }
I20260928 08:46:58.212316 13322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a [sys.catalog]: This master's current role is: LEADER
I20260928 08:46:58.212816 13321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2e6f3e4dd306497581f87a1072af880a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e6f3e4dd306497581f87a1072af880a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34293 } } }
I20260928 08:46:58.213090 13321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a [sys.catalog]: This master's current role is: LEADER
I20260928 08:46:58.221194 13326 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:46:58.227829 13326 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:46:58.229583 13326 catalog_manager.cc:1259] Loaded cluster ID: 51e53e40f9da451cba3ed97ef493dcb0
I20260928 08:46:58.229770 13326 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:46:58.234951 13326 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:46:58.236835 13326 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a: Loaded TSK: 0
I20260928 08:46:58.238732 13326 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:46:58.241173 13316 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260928 08:46:58.595656 13318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:46:58.596118 13318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:46:58.596318 13318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:46:58.608350 13318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:46:58.608498 13318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:46:58.608750 13318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:46:58.624413 13318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:38227/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:38533
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=40335
--tserver_master_addrs=127.0.97.62:34293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:46:58.626122 13318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:46:58.627915 13318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:46:58.665782 13353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:46:58.668655 13354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:46:58.676702 13318 server_base.cc:1034] running on GCE node
W20260928 08:46:58.677522 13356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:46:58.678494 13318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:46:58.684466 13318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:46:58.686388 13318 hybrid_clock.cc:648] HybridClock initialized: now 1790585218686312 us; error 85 us; skew 500 ppm
I20260928 08:46:58.686872 13318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:46:58 dist-test-slave-n6xs krb5kdc[12270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:46:58.691387 13318 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:46:58.705566 13318 webserver.cc:460] Webserver started at http://127.0.97.1:40335/ using document root <none> and password file <none>
I20260928 08:46:58.706684 13318 fs_manager.cc:359] Metadata directory not provided
I20260928 08:46:58.706933 13318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 08:46:58 dist-test-slave-n6xs krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 08:46:58.740197 13318 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.003s sys 0.007s
I20260928 08:46:58.750339 13365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:46:58.752466 13318 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.003s sys 0.007s
I20260928 08:46:58.752754 13318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-0/wal
uuid: "5acbab1c85d348bca5d92cee3682e166"
format_stamp: "Formatted at 2026-09-28 08:46:55 on dist-test-slave-n6xs"
server_key: "f8ef3427642d4caf506742e24e7932e8"
server_key_iv: "9fa2b721c2cb54c13fb57c6285d10e3b"
server_key_version: "kuduclusterkey@0"
I20260928 08:46:58.754050 13318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:46:58.806789 13318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:46:58.813627 13318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:46:58.815945 13318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:46:58.827085 13318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:46:58.831053 13318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:46:58.831195 13318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:46:58.831346 13318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:46:58.831449 13318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:46:58.918792 13318 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38533
I20260928 08:46:58.918848 13477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38533 every 8 connection(s)
I20260928 08:46:58.921012 13318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:46:58.924388 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 13318
I20260928 08:46:58.925395 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:38227/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:36277
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:34293
--webserver_port=45017
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:38227/kms with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:46:58 dist-test-slave-n6xs krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:46:58.975929 13480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:58.931658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34293 (local address 127.0.97.1:39311)
0928 08:46:58.941013 (+ 9355us) negotiation.cc:107] Waiting for socket to connect
0928 08:46:58.941069 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 08:46:58.944334 (+ 3265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:46:58.948153 (+ 3819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:46:58.948167 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:46:58.948853 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:46:58.949659 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:46:58.949686 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:46:58.952340 (+ 2654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:46:58.952348 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:46:58.954586 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:46:58.954604 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:46:58.956535 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:46:58.956544 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:46:58.956845 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:58.962270 (+ 5425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:46:58.962307 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:46:58.966619 (+ 4312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:46:58.971029 (+ 4410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:46:58.971040 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:46:58.971058 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:46:58.971681 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:46:58.972294 (+ 613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:46:58.972299 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:46:58.972303 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:46:58.972526 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:46:58.973635 (+ 1109us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:46:58.973643 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:46:58.974064 (+ 421us) client_negotiation.cc:769] Sending connection context
0928 08:46:58.974398 (+ 334us) client_negotiation.cc:240] Negotiation successful
0928 08:46:58.974843 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9068,"thread_start_us":8812,"threads_started":1}
I20260928 08:46:58.978121 13481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:58.932825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:39311 (local address 127.0.97.62:34293)
0928 08:46:58.940377 (+ 7552us) server_negotiation.cc:206] Beginning negotiation
0928 08:46:58.940404 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 08:46:58.941569 (+ 1165us) server_negotiation.cc:407] Connection header received
0928 08:46:58.945238 (+ 3669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:46:58.945262 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:46:58.945791 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:46:58.946204 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:46:58.949906 (+ 3702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:58.952121 (+ 2215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:58.954789 (+ 2668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:58.955928 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:58.956127 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:58.966982 (+ 10855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:46:58.966994 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:46:58.967016 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:46:58.967075 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:46:58.970855 (+ 3780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:46:58.971896 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:46:58.971906 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:46:58.971915 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:46:58.972127 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:46:58.972664 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:46:58.972670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:46:58.972673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:46:58.973348 (+ 675us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:46:58.973542 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 08:46:58.976928 (+ 3386us) server_negotiation.cc:299] Negotiation successful
0928 08:46:58.977244 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7277,"thread_start_us":7063,"threads_started":1}
I20260928 08:46:58.981341 13478 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34293
I20260928 08:46:58.981673 13478 heartbeater.cc:464] Registering TS with master...
I20260928 08:46:58.982532 13478 heartbeater.cc:511] Master 127.0.97.62:34293 requested a full tablet report, sending...
W20260928 08:46:58.985623 13478 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:34293 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 08:46:58.985843 13478 heartbeater.cc:464] Registering TS with master...
I20260928 08:46:58.986114 13478 heartbeater.cc:511] Master 127.0.97.62:34293 requested a full tablet report, sending...
I20260928 08:46:58.987221 13478 heartbeater.cc:464] Registering TS with master...
I20260928 08:46:58.987479 13478 heartbeater.cc:511] Master 127.0.97.62:34293 requested a full tablet report, sending...
W20260928 08:46:58.988516 13478 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 08:46:59.378714 13482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:46:59.379132 13482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:46:59.379375 13482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:46:59.391136 13482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:46:59.391305 13482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:46:59.391551 13482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:46:59.407663 13482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:38227/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:36277
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=45017
--tserver_master_addrs=127.0.97.62:34293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:46:59.409758 13482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:46:59.411662 13482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 28 08:46:59 dist-test-slave-n6xs krb5kdc[12270](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790585219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 08:46:59.446751 13498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:46:59.447745 13499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:46:59.459730 13501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:46:59.460315 13482 server_base.cc:1034] running on GCE node
I20260928 08:46:59.461067 13482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:46:59.462250 13482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:46:59.463529 13482 hybrid_clock.cc:648] HybridClock initialized: now 1790585219463459 us; error 63 us; skew 500 ppm
I20260928 08:46:59.463934 13482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:46:59 dist-test-slave-n6xs krb5kdc[12270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:46:59.467186 13482 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:46:59.469074 13482 webserver.cc:460] Webserver started at http://127.0.97.2:45017/ using document root <none> and password file <none>
I20260928 08:46:59.469771 13482 fs_manager.cc:359] Metadata directory not provided
I20260928 08:46:59.469924 13482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 08:46:59 dist-test-slave-n6xs krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 08:46:59.507615 13482 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 08:46:59.511237 13509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:46:59.513087 13482 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 08:46:59.513304 13482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-1/wal
uuid: "24be679363434337b186be2bbded83e9"
format_stamp: "Formatted at 2026-09-28 08:46:56 on dist-test-slave-n6xs"
server_key: "ee2c6263b1eee0f59699988e786a392e"
server_key_iv: "3c2b8240a4e3db2d98950b3ed2598224"
server_key_version: "kuduclusterkey@0"
I20260928 08:46:59.514386 13482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:46:59.556114 13482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:46:59.560936 13482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:46:59.561239 13482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:46:59.562336 13482 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:46:59.563855 13482 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:46:59.563952 13482 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:46:59.564101 13482 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:46:59.564174 13482 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:46:59.605561 13482 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:36277
I20260928 08:46:59.605603 13622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:36277 every 8 connection(s)
I20260928 08:46:59.606984 13482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:46:59.609840 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 13482
I20260928 08:46:59.610803 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.97.20:38227/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:38361
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:34293
--webserver_port=34379
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:38227/kms with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:46:59 dist-test-slave-n6xs krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:46:59.646708 13626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:59.610484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34293 (local address 127.0.97.2:39691)
0928 08:46:59.620197 (+ 9713us) negotiation.cc:107] Waiting for socket to connect
0928 08:46:59.620251 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 08:46:59.623961 (+ 3710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:46:59.626088 (+ 2127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:46:59.626103 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:46:59.626733 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:46:59.627590 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:46:59.627632 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:46:59.629179 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:46:59.629184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:46:59.631383 (+ 2199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:46:59.631398 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:46:59.633420 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:46:59.633427 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:46:59.633730 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:59.634564 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:46:59.634600 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:46:59.637551 (+ 2951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:46:59.641792 (+ 4241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:46:59.641801 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:46:59.641821 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:46:59.642404 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:46:59.642872 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:46:59.642878 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:46:59.642881 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:46:59.643117 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:46:59.643899 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:46:59.643907 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:46:59.644349 (+ 442us) client_negotiation.cc:769] Sending connection context
0928 08:46:59.644686 (+ 337us) client_negotiation.cc:240] Negotiation successful
0928 08:46:59.645265 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9370,"thread_start_us":10402,"threads_started":1}
I20260928 08:46:59.647879 13625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:46:59.609850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:39691 (local address 127.0.97.62:34293)
0928 08:46:59.625299 (+ 15449us) server_negotiation.cc:206] Beginning negotiation
0928 08:46:59.625306 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:46:59.625327 (+ 21us) server_negotiation.cc:407] Connection header received
0928 08:46:59.625392 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:46:59.625397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:46:59.625489 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:46:59.625623 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:46:59.627839 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:59.628989 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:59.631580 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:46:59.632213 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:46:59.632333 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:46:59.637838 (+ 5505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:46:59.637845 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:46:59.637850 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:46:59.637888 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:46:59.641622 (+ 3734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:46:59.642574 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:46:59.642581 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:46:59.642585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:46:59.642733 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:46:59.643264 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:46:59.643272 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:46:59.643275 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:46:59.643643 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:46:59.643779 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 08:46:59.647208 (+ 3429us) server_negotiation.cc:299] Negotiation successful
0928 08:46:59.647462 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15309,"thread_start_us":15194,"threads_started":1}
I20260928 08:46:59.650064 13623 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34293
I20260928 08:46:59.650388 13623 heartbeater.cc:464] Registering TS with master...
I20260928 08:46:59.651227 13623 heartbeater.cc:511] Master 127.0.97.62:34293 requested a full tablet report, sending...
I20260928 08:46:59.654085 13281 ts_manager.cc:194] Registered new tserver with Master: 24be679363434337b186be2bbded83e9 (127.0.97.2:36277)
I20260928 08:46:59.656358 13281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:39691
Sep 28 08:46:59 dist-test-slave-n6xs krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 08:46:59.988653 13478 heartbeater.cc:464] Registering TS with master...
I20260928 08:46:59.989097 13478 heartbeater.cc:511] Master 127.0.97.62:34293 requested a full tablet report, sending...
I20260928 08:46:59.990392 13281 ts_manager.cc:194] Registered new tserver with Master: 5acbab1c85d348bca5d92cee3682e166 (127.0.97.1:38533)
I20260928 08:46:59.991604 13281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:39311
W20260928 08:47:00.081574 13627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:47:00.082068 13627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:47:00.082415 13627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:47:00.094240 13627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:47:00.094441 13627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:47:00.094666 13627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:47:00.111167 13627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.97.20:38227/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:38361
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=34379
--tserver_master_addrs=127.0.97.62:34293
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:47:00.113113 13627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:47:00.114897 13627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:47:00.141407 13632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:00.145478 13633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:47:00.153533 13627 server_base.cc:1034] running on GCE node
W20260928 08:47:00.153967 13635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:47:00.154768 13627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:47:00.159371 13627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:47:00.160676 13627 hybrid_clock.cc:648] HybridClock initialized: now 1790585220160609 us; error 57 us; skew 500 ppm
I20260928 08:47:00.161149 13627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:47:00 dist-test-slave-n6xs krb5kdc[12270](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:47:00.165592 13627 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:47:00.169389 13627 webserver.cc:460] Webserver started at http://127.0.97.3:34379/ using document root <none> and password file <none>
I20260928 08:47:00.170359 13627 fs_manager.cc:359] Metadata directory not provided
I20260928 08:47:00.170534 13627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 08:47:00 dist-test-slave-n6xs krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 08:47:00.197001 13627 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 08:47:00.204511 13643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:00.206456 13627 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.007s sys 0.000s
I20260928 08:47:00.206699 13627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-2/wal
uuid: "d6b545ca970645febef66643334ce4ae"
format_stamp: "Formatted at 2026-09-28 08:46:57 on dist-test-slave-n6xs"
server_key: "00c8d55f827f4f097658856ccc346c63"
server_key_iv: "f1d13f52921a5c3b44664ca7db58c130"
server_key_version: "kuduclusterkey@0"
I20260928 08:47:00.208211 13627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:47:00.244015 13627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
Sep 28 08:47:00 dist-test-slave-n6xs krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for HTTP/127.0.97.20@KRBTEST.COM
I20260928 08:47:00.251119 13627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:47:00.251735 13627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:47:00.258441 13627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:47:00.279095 13627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:47:00.279304 13627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 08:47:00.279458 13627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:47:00.279577 13627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:00.505322 13627 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:38361
I20260928 08:47:00.505326 13756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:38361 every 8 connection(s)
I20260928 08:47:00.507527 13627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:47:00.513695 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 13627
Sep 28 08:47:00 dist-test-slave-n6xs krb5kdc[12270](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:47:00.594483 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:00.531320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50989 (local address 127.0.97.62:34293)
0928 08:47:00.572681 (+ 41361us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:00.572688 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:00.572710 (+ 22us) server_negotiation.cc:407] Connection header received
0928 08:47:00.572770 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:00.572777 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:00.572871 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:00.573023 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:00.575863 (+ 2840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:00.577519 (+ 1656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:00.580838 (+ 3319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:00.581765 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:00.581910 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:00.586942 (+ 5032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:00.586950 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:00.586956 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:00.587006 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:00.590849 (+ 3843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:00.591843 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:00.591850 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:00.591856 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:00.591992 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:00.592526 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:00.592531 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:00.592534 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:00.592850 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:00.592999 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:00.593877 (+ 878us) server_negotiation.cc:299] Negotiation successful
0928 08:47:00.594084 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41215,"thread_start_us":32454,"threads_started":1}
I20260928 08:47:00.595551 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:00.532541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34293 (local address 127.0.97.3:50989)
0928 08:47:00.560392 (+ 27851us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:00.560456 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:00.563264 (+ 2808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:00.573450 (+ 10186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:00.573480 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:00.574364 (+ 884us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:47:00.575515 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:00.575548 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:00.577780 (+ 2232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:00.577789 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:00.580612 (+ 2823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:00.580633 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:00.582285 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:00.582295 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:00.582728 (+ 433us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:00.583745 (+ 1017us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:47:00.583779 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:47:00.586664 (+ 2885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:47:00.591024 (+ 4360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:00.591033 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:00.591054 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:00.591666 (+ 612us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:00.592126 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:00.592133 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:00.592137 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:00.592371 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:00.592999 (+ 628us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:47:00.593008 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:47:00.593447 (+ 439us) client_negotiation.cc:769] Sending connection context
0928 08:47:00.593785 (+ 338us) client_negotiation.cc:240] Negotiation successful
0928 08:47:00.594332 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27484,"thread_start_us":31230,"threads_started":1}
I20260928 08:47:00.597126 13757 heartbeater.cc:347] Connected to a master server at 127.0.97.62:34293
I20260928 08:47:00.597452 13757 heartbeater.cc:464] Registering TS with master...
I20260928 08:47:00.598245 13757 heartbeater.cc:511] Master 127.0.97.62:34293 requested a full tablet report, sending...
I20260928 08:47:00.599797 13281 ts_manager.cc:194] Registered new tserver with Master: d6b545ca970645febef66643334ce4ae (127.0.97.3:38361)
I20260928 08:47:00.600666 13281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50989
I20260928 08:47:00.602756 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:00.514895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53164 (local address 127.0.97.62:34293)
0928 08:47:00.559688 (+ 44793us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:00.559696 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:00.565575 (+ 5879us) server_negotiation.cc:407] Connection header received
0928 08:47:00.565665 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:00.565672 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:00.565791 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:00.565912 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:00.573896 (+ 7984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:00.575110 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:00.577178 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:00.577881 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:00.578006 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:00.587792 (+ 9786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:00.587801 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:00.587849 (+ 48us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:00.587913 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:00.591796 (+ 3883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:00.592720 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:00.592728 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:00.592732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:00.592924 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:00.600935 (+ 8011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:00.600945 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:00.600949 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:00.601357 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:00.601670 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:00.602044 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 08:47:00.602306 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44625,"thread_start_us":57453,"threads_started":1}
I20260928 08:47:00.606307 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:47:00.627184 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:00.609901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53166 (local address 127.0.97.62:34293)
0928 08:47:00.612020 (+ 2119us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:00.612024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:00.613148 (+ 1124us) server_negotiation.cc:407] Connection header received
0928 08:47:00.613492 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:00.613514 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:00.613597 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:00.613729 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:00.615964 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:00.616955 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:00.618393 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:00.618845 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:00.618986 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:00.621743 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:00.621748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:00.621752 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:00.621782 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:00.624607 (+ 2825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:00.625280 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:00.625290 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:00.625292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:00.625411 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:00.625791 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:00.625796 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:00.625799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:00.626067 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:00.626307 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:00.626626 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 08:47:00.626854 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2008}
I20260928 08:47:00.632289 13281 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:47:00.639925 13281 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:47:00.660858 13623 heartbeater.cc:503] Master 127.0.97.62:34293 was elected leader, sending a full tablet report...
I20260928 08:47:00.692271 13780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:00.654994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:38361 (local address 127.0.0.1:34248)
0928 08:47:00.667491 (+ 12497us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:00.667547 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:00.667755 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:00.682660 (+ 14905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:00.682672 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:00.682715 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:47:00.683237 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:00.683574 (+ 337us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:00.687524 (+ 3950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:00.687531 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:00.689866 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:00.689879 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:00.691220 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:00.691226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:00.691434 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:00.691489 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 08:47:00.691636 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 08:47:00.691864 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12180,"thread_start_us":17016,"threads_started":1}
I20260928 08:47:00.693091 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:00.655697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34248 (local address 127.0.97.3:38361)
0928 08:47:00.681410 (+ 25713us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:00.681423 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:00.681478 (+ 55us) server_negotiation.cc:407] Connection header received
0928 08:47:00.681604 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:00.681616 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:00.681967 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:00.682227 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:47:00.683841 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:00.686115 (+ 2274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:00.690043 (+ 3928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:00.690996 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:00.691163 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:00.691402 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:00.692570 (+ 1168us) server_negotiation.cc:299] Negotiation successful
0928 08:47:00.692743 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25474,"thread_start_us":27597,"threads_started":1}
I20260928 08:47:00.697193 13692 tablet_service.cc:1467] Processing CreateTablet for tablet 5f0bde561a284eb6abd6bf628ba2aace (DEFAULT_TABLE table=test-table [id=0a422b3e2b1043eda593909484088c78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:47:00.698788 13692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f0bde561a284eb6abd6bf628ba2aace. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:47:00.699227 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:00.655943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58162 (local address 127.0.97.1:38533)
0928 08:47:00.686714 (+ 30771us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:00.686721 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:00.686745 (+ 24us) server_negotiation.cc:407] Connection header received
0928 08:47:00.686819 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:00.686825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:00.687012 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:00.687220 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:47:00.691524 (+ 4304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:00.694280 (+ 2756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:00.696740 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:00.698188 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:00.698334 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:00.698650 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:00.698731 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 08:47:00.698900 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30602,"thread_start_us":30417,"threads_started":1}
I20260928 08:47:00.699260 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:00.655161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38533 (local address 127.0.0.1:58162)
0928 08:47:00.672289 (+ 17128us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:00.672308 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:00.672437 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:00.687204 (+ 14767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:00.687208 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:00.687233 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:47:00.687601 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:00.687616 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:00.694435 (+ 6819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:00.694441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:00.696533 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:00.696552 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:00.698402 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:00.698408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:00.698573 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:00.698608 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 08:47:00.698695 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 08:47:00.698861 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16994,"mutex_wait_us":5,"thread_start_us":12837,"threads_started":1}
I20260928 08:47:00.702400 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:00.672740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36277 (local address 127.0.0.1:51930)
0928 08:47:00.679931 (+ 7191us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:00.679955 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:00.680099 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:00.692122 (+ 12023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:00.692127 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:00.692156 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:47:00.692589 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:00.692603 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:00.695492 (+ 2889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:00.695497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:00.697560 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:00.697575 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:00.701570 (+ 3995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:00.701580 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:00.701790 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:00.701827 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 08:47:00.701913 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 08:47:00.702047 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7015,"thread_start_us":7448,"threads_started":1}
I20260928 08:47:00.702908 13413 tablet_service.cc:1467] Processing CreateTablet for tablet 5f0bde561a284eb6abd6bf628ba2aace (DEFAULT_TABLE table=test-table [id=0a422b3e2b1043eda593909484088c78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:47:00.703799 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:00.672887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51930 (local address 127.0.97.2:36277)
0928 08:47:00.691233 (+ 18346us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:00.691244 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:00.691275 (+ 31us) server_negotiation.cc:407] Connection header received
0928 08:47:00.691377 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:00.691385 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:00.691649 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:00.693067 (+ 1418us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:47:00.693429 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:00.695335 (+ 1906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:00.699781 (+ 4446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:00.701298 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:00.702869 (+ 1571us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:00.703116 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:00.703188 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 08:47:00.703353 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18158,"thread_start_us":18964,"threads_started":1}
I20260928 08:47:00.704391 13413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f0bde561a284eb6abd6bf628ba2aace. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:47:00.707520 13558 tablet_service.cc:1467] Processing CreateTablet for tablet 5f0bde561a284eb6abd6bf628ba2aace (DEFAULT_TABLE table=test-table [id=0a422b3e2b1043eda593909484088c78]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:47:00.709604 13558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f0bde561a284eb6abd6bf628ba2aace. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:47:00.721800 13785 tablet_bootstrap.cc:492] T 5f0bde561a284eb6abd6bf628ba2aace P d6b545ca970645febef66643334ce4ae: Bootstrap starting.
I20260928 08:47:00.727543 13785 tablet_bootstrap.cc:654] T 5f0bde561a284eb6abd6bf628ba2aace P d6b545ca970645febef66643334ce4ae: Neither blocks nor log segments found. Creating new log.
I20260928 08:47:00.728462 13786 tablet_bootstrap.cc:492] T 5f0bde561a284eb6abd6bf628ba2aace P 5acbab1c85d348bca5d92cee3682e166: Bootstrap starting.
I20260928 08:47:00.732574 13785 log.cc:824] T 5f0bde561a284eb6abd6bf628ba2aace P d6b545ca970645febef66643334ce4ae: Log is configured to *not* fsync() on all Append() calls
I20260928 08:47:00.733114 13786 tablet_bootstrap.cc:654] T 5f0bde561a284eb6abd6bf628ba2aace P 5acbab1c85d348bca5d92cee3682e166: Neither blocks nor log segments found. Creating new log.
I20260928 08:47:00.734767 13786 log.cc:824] T 5f0bde561a284eb6abd6bf628ba2aace P 5acbab1c85d348bca5d92cee3682e166: Log is configured to *not* fsync() on all Append() calls
I20260928 08:47:00.736703 13787 tablet_bootstrap.cc:492] T 5f0bde561a284eb6abd6bf628ba2aace P 24be679363434337b186be2bbded83e9: Bootstrap starting.
I20260928 08:47:00.742376 13785 tablet_bootstrap.cc:492] T 5f0bde561a284eb6abd6bf628ba2aace P d6b545ca970645febef66643334ce4ae: No bootstrap required, opened a new log
I20260928 08:47:00.742405 13786 tablet_bootstrap.cc:492] T 5f0bde561a284eb6abd6bf628ba2aace P 5acbab1c85d348bca5d92cee3682e166: No bootstrap required, opened a new log
I20260928 08:47:00.742393 13787 tablet_bootstrap.cc:654] T 5f0bde561a284eb6abd6bf628ba2aace P 24be679363434337b186be2bbded83e9: Neither blocks nor log segments found. Creating new log.
I20260928 08:47:00.742689 13786 ts_tablet_manager.cc:1397] T 5f0bde561a284eb6abd6bf628ba2aace P 5acbab1c85d348bca5d92cee3682e166: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.007s
I20260928 08:47:00.742698 13785 ts_tablet_manager.cc:1397] T 5f0bde561a284eb6abd6bf628ba2aace P d6b545ca970645febef66643334ce4ae: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.005s
I20260928 08:47:00.743875 13787 log.cc:824] T 5f0bde561a284eb6abd6bf628ba2aace P 24be679363434337b186be2bbded83e9: Log is configured to *not* fsync() on all Append() calls
I20260928 08:47:00.753427 13785 raft_consensus.cc:348] T 5f0bde561a284eb6abd6bf628ba2aace P d6b545ca970645febef66643334ce4ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b545ca970645febef66643334ce4ae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38361 } } peers { permanent_uuid: "5acbab1c85d348bca5d92cee3682e166" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38533 } } peers { permanent_uuid: "24be679363434337b186be2bbded83e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36277 } }
I20260928 08:47:00.753988 13785 raft_consensus.cc:374] T 5f0bde561a284eb6abd6bf628ba2aace P d6b545ca970645febef66643334ce4ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:47:00.754138 13785 raft_consensus.cc:729] T 5f0bde561a284eb6abd6bf628ba2aace P d6b545ca970645febef66643334ce4ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b545ca970645febef66643334ce4ae, State: Initialized, Role: FOLLOWER
I20260928 08:47:00.754875 13785 consensus_queue.cc:260] T 5f0bde561a284eb6abd6bf628ba2aace P d6b545ca970645febef66643334ce4ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b545ca970645febef66643334ce4ae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38361 } } peers { permanent_uuid: "5acbab1c85d348bca5d92cee3682e166" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38533 } } peers { permanent_uuid: "24be679363434337b186be2bbded83e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36277 } }
I20260928 08:47:00.757364 13787 tablet_bootstrap.cc:492] T 5f0bde561a284eb6abd6bf628ba2aace P 24be679363434337b186be2bbded83e9: No bootstrap required, opened a new log
I20260928 08:47:00.757763 13787 ts_tablet_manager.cc:1397] T 5f0bde561a284eb6abd6bf628ba2aace P 24be679363434337b186be2bbded83e9: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.011s
I20260928 08:47:00.757889 13786 raft_consensus.cc:348] T 5f0bde561a284eb6abd6bf628ba2aace P 5acbab1c85d348bca5d92cee3682e166 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b545ca970645febef66643334ce4ae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38361 } } peers { permanent_uuid: "5acbab1c85d348bca5d92cee3682e166" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38533 } } peers { permanent_uuid: "24be679363434337b186be2bbded83e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36277 } }
I20260928 08:47:00.758491 13786 raft_consensus.cc:374] T 5f0bde561a284eb6abd6bf628ba2aace P 5acbab1c85d348bca5d92cee3682e166 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:47:00.758653 13786 raft_consensus.cc:729] T 5f0bde561a284eb6abd6bf628ba2aace P 5acbab1c85d348bca5d92cee3682e166 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5acbab1c85d348bca5d92cee3682e166, State: Initialized, Role: FOLLOWER
I20260928 08:47:00.759495 13786 consensus_queue.cc:260] T 5f0bde561a284eb6abd6bf628ba2aace P 5acbab1c85d348bca5d92cee3682e166 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b545ca970645febef66643334ce4ae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38361 } } peers { permanent_uuid: "5acbab1c85d348bca5d92cee3682e166" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38533 } } peers { permanent_uuid: "24be679363434337b186be2bbded83e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36277 } }
I20260928 08:47:00.763195 13757 heartbeater.cc:503] Master 127.0.97.62:34293 was elected leader, sending a full tablet report...
I20260928 08:47:00.763296 13785 ts_tablet_manager.cc:1428] T 5f0bde561a284eb6abd6bf628ba2aace P d6b545ca970645febef66643334ce4ae: Time spent starting tablet: real 0.020s user 0.012s sys 0.007s
I20260928 08:47:00.764979 13478 heartbeater.cc:503] Master 127.0.97.62:34293 was elected leader, sending a full tablet report...
I20260928 08:47:00.765985 13786 ts_tablet_manager.cc:1428] T 5f0bde561a284eb6abd6bf628ba2aace P 5acbab1c85d348bca5d92cee3682e166: Time spent starting tablet: real 0.023s user 0.012s sys 0.007s
I20260928 08:47:00.771004 13787 raft_consensus.cc:348] T 5f0bde561a284eb6abd6bf628ba2aace P 24be679363434337b186be2bbded83e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b545ca970645febef66643334ce4ae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38361 } } peers { permanent_uuid: "5acbab1c85d348bca5d92cee3682e166" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38533 } } peers { permanent_uuid: "24be679363434337b186be2bbded83e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36277 } }
I20260928 08:47:00.771421 13787 raft_consensus.cc:374] T 5f0bde561a284eb6abd6bf628ba2aace P 24be679363434337b186be2bbded83e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:47:00.771554 13787 raft_consensus.cc:729] T 5f0bde561a284eb6abd6bf628ba2aace P 24be679363434337b186be2bbded83e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24be679363434337b186be2bbded83e9, State: Initialized, Role: FOLLOWER
I20260928 08:47:00.772195 13787 consensus_queue.cc:260] T 5f0bde561a284eb6abd6bf628ba2aace P 24be679363434337b186be2bbded83e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b545ca970645febef66643334ce4ae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38361 } } peers { permanent_uuid: "5acbab1c85d348bca5d92cee3682e166" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38533 } } peers { permanent_uuid: "24be679363434337b186be2bbded83e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36277 } }
I20260928 08:47:00.773802 13787 ts_tablet_manager.cc:1428] T 5f0bde561a284eb6abd6bf628ba2aace P 24be679363434337b186be2bbded83e9: Time spent starting tablet: real 0.016s user 0.013s sys 0.000s
W20260928 08:47:00.834725 13758 tablet.cc:2369] T 5f0bde561a284eb6abd6bf628ba2aace P d6b545ca970645febef66643334ce4ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:47:00.838326 13791 raft_consensus.cc:482] T 5f0bde561a284eb6abd6bf628ba2aace P d6b545ca970645febef66643334ce4ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:47:00.838601 13791 raft_consensus.cc:504] T 5f0bde561a284eb6abd6bf628ba2aace P d6b545ca970645febef66643334ce4ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b545ca970645febef66643334ce4ae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38361 } } peers { permanent_uuid: "5acbab1c85d348bca5d92cee3682e166" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38533 } } peers { permanent_uuid: "24be679363434337b186be2bbded83e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36277 } }
I20260928 08:47:00.840353 13791 leader_election.cc:288] T 5f0bde561a284eb6abd6bf628ba2aace P d6b545ca970645febef66643334ce4ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5acbab1c85d348bca5d92cee3682e166 (127.0.97.1:38533), 24be679363434337b186be2bbded83e9 (127.0.97.2:36277)
I20260928 08:47:00.847215 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:00.840804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48563 (local address 127.0.97.1:38533)
0928 08:47:00.841048 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:00.841056 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:00.841077 (+ 21us) server_negotiation.cc:407] Connection header received
0928 08:47:00.841372 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:00.841379 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:00.841471 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:00.841554 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:47:00.842111 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:00.843052 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:00.845171 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:00.846263 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:00.846401 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:00.846513 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:00.846841 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 08:47:00.846975 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260928 08:47:00.847239 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:00.840712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38533 (local address 127.0.97.3:48563)
0928 08:47:00.840995 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:00.841018 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:00.841162 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:00.841732 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:00.841736 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:00.841760 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:47:00.841973 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:00.841981 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:00.843195 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:00.843201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:00.844972 (+ 1771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:00.844988 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:00.846444 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:00.846449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:00.846663 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:00.846700 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 08:47:00.846798 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 08:47:00.846933 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":51}
I20260928 08:47:00.848114 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:00.841403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38549 (local address 127.0.97.2:36277)
0928 08:47:00.841605 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:00.841610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:00.841702 (+ 92us) server_negotiation.cc:407] Connection header received
0928 08:47:00.841946 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:00.841951 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:00.842027 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:00.842118 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:47:00.842687 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:00.843906 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:00.845896 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:00.846895 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:00.847010 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:00.847111 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:00.847728 (+ 617us) server_negotiation.cc:299] Negotiation successful
0928 08:47:00.847885 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 08:47:00.848196 13794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:00.840796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:36277 (local address 127.0.97.3:38549)
0928 08:47:00.841602 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:00.841619 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:00.841784 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:00.842140 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:00.842155 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:00.842185 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:47:00.842522 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:00.842532 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:00.844043 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:00.844049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:00.845747 (+ 1698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:00.845757 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:00.847257 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:00.847266 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:00.847452 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:00.847483 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 08:47:00.847680 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 08:47:00.847806 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":32,"thread_start_us":806,"threads_started":1}
I20260928 08:47:00.848223 13433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0bde561a284eb6abd6bf628ba2aace" candidate_uuid: "d6b545ca970645febef66643334ce4ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5acbab1c85d348bca5d92cee3682e166" is_pre_election: true
I20260928 08:47:00.848820 13433 raft_consensus.cc:2454] T 5f0bde561a284eb6abd6bf628ba2aace P 5acbab1c85d348bca5d92cee3682e166 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6b545ca970645febef66643334ce4ae in term 0.
I20260928 08:47:00.849490 13578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0bde561a284eb6abd6bf628ba2aace" candidate_uuid: "d6b545ca970645febef66643334ce4ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24be679363434337b186be2bbded83e9" is_pre_election: true
I20260928 08:47:00.849598 13644 leader_election.cc:302] T 5f0bde561a284eb6abd6bf628ba2aace P d6b545ca970645febef66643334ce4ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5acbab1c85d348bca5d92cee3682e166, d6b545ca970645febef66643334ce4ae; no voters:
I20260928 08:47:00.850013 13791 raft_consensus.cc:2781] T 5f0bde561a284eb6abd6bf628ba2aace P d6b545ca970645febef66643334ce4ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:47:00.850013 13578 raft_consensus.cc:2454] T 5f0bde561a284eb6abd6bf628ba2aace P 24be679363434337b186be2bbded83e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6b545ca970645febef66643334ce4ae in term 0.
I20260928 08:47:00.850171 13791 raft_consensus.cc:482] T 5f0bde561a284eb6abd6bf628ba2aace P d6b545ca970645febef66643334ce4ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:47:00.850304 13791 raft_consensus.cc:3037] T 5f0bde561a284eb6abd6bf628ba2aace P d6b545ca970645febef66643334ce4ae [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:47:00.852630 13791 raft_consensus.cc:504] T 5f0bde561a284eb6abd6bf628ba2aace P d6b545ca970645febef66643334ce4ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b545ca970645febef66643334ce4ae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38361 } } peers { permanent_uuid: "5acbab1c85d348bca5d92cee3682e166" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38533 } } peers { permanent_uuid: "24be679363434337b186be2bbded83e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36277 } }
I20260928 08:47:00.853232 13791 leader_election.cc:288] T 5f0bde561a284eb6abd6bf628ba2aace P d6b545ca970645febef66643334ce4ae [CANDIDATE]: Term 1 election: Requested vote from peers 5acbab1c85d348bca5d92cee3682e166 (127.0.97.1:38533), 24be679363434337b186be2bbded83e9 (127.0.97.2:36277)
I20260928 08:47:00.853693 13433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0bde561a284eb6abd6bf628ba2aace" candidate_uuid: "d6b545ca970645febef66643334ce4ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5acbab1c85d348bca5d92cee3682e166"
I20260928 08:47:00.853757 13578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0bde561a284eb6abd6bf628ba2aace" candidate_uuid: "d6b545ca970645febef66643334ce4ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24be679363434337b186be2bbded83e9"
I20260928 08:47:00.853899 13433 raft_consensus.cc:3037] T 5f0bde561a284eb6abd6bf628ba2aace P 5acbab1c85d348bca5d92cee3682e166 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:47:00.853955 13578 raft_consensus.cc:3037] T 5f0bde561a284eb6abd6bf628ba2aace P 24be679363434337b186be2bbded83e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:47:00.856345 13433 raft_consensus.cc:2454] T 5f0bde561a284eb6abd6bf628ba2aace P 5acbab1c85d348bca5d92cee3682e166 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6b545ca970645febef66643334ce4ae in term 1.
I20260928 08:47:00.856712 13644 leader_election.cc:302] T 5f0bde561a284eb6abd6bf628ba2aace P d6b545ca970645febef66643334ce4ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5acbab1c85d348bca5d92cee3682e166, d6b545ca970645febef66643334ce4ae; no voters:
I20260928 08:47:00.856980 13791 raft_consensus.cc:2781] T 5f0bde561a284eb6abd6bf628ba2aace P d6b545ca970645febef66643334ce4ae [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:47:00.856973 13578 raft_consensus.cc:2454] T 5f0bde561a284eb6abd6bf628ba2aace P 24be679363434337b186be2bbded83e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6b545ca970645febef66643334ce4ae in term 1.
I20260928 08:47:00.857825 13791 raft_consensus.cc:686] T 5f0bde561a284eb6abd6bf628ba2aace P d6b545ca970645febef66643334ce4ae [term 1 LEADER]: Becoming Leader. State: Replica: d6b545ca970645febef66643334ce4ae, State: Running, Role: LEADER
I20260928 08:47:00.858388 13791 consensus_queue.cc:237] T 5f0bde561a284eb6abd6bf628ba2aace P d6b545ca970645febef66643334ce4ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b545ca970645febef66643334ce4ae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38361 } } peers { permanent_uuid: "5acbab1c85d348bca5d92cee3682e166" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38533 } } peers { permanent_uuid: "24be679363434337b186be2bbded83e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36277 } }
W20260928 08:47:00.860235 13624 tablet.cc:2369] T 5f0bde561a284eb6abd6bf628ba2aace P 24be679363434337b186be2bbded83e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:47:00.863438 13281 catalog_manager.cc:5495] T 5f0bde561a284eb6abd6bf628ba2aace P d6b545ca970645febef66643334ce4ae reported cstate change: term changed from 0 to 1, leader changed from <none> to d6b545ca970645febef66643334ce4ae (127.0.97.3). New cstate: current_term: 1 leader_uuid: "d6b545ca970645febef66643334ce4ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b545ca970645febef66643334ce4ae" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 38361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5acbab1c85d348bca5d92cee3682e166" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24be679363434337b186be2bbded83e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36277 } health_report { overall_health: UNKNOWN } } }
I20260928 08:47:00.885414 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:00.880030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34254 (local address 127.0.97.3:38361)
0928 08:47:00.880245 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:00.880251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:00.880367 (+ 116us) server_negotiation.cc:407] Connection header received
0928 08:47:00.880627 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:00.880631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:00.880723 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:00.880874 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:47:00.881252 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:00.881958 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:00.883278 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:00.883677 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:00.883841 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:00.884330 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:47:00.884845 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:47:00.885008 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:00.885062 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 08:47:00.885140 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 08:47:00.895015 13433 raft_consensus.cc:1261] T 5f0bde561a284eb6abd6bf628ba2aace P 5acbab1c85d348bca5d92cee3682e166 [term 1 FOLLOWER]: Refusing update from remote peer d6b545ca970645febef66643334ce4ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:47:00.895150 13578 raft_consensus.cc:1261] T 5f0bde561a284eb6abd6bf628ba2aace P 24be679363434337b186be2bbded83e9 [term 1 FOLLOWER]: Refusing update from remote peer d6b545ca970645febef66643334ce4ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:47:00.896174 13795 consensus_queue.cc:1035] T 5f0bde561a284eb6abd6bf628ba2aace P d6b545ca970645febef66643334ce4ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "5acbab1c85d348bca5d92cee3682e166" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:47:00.896535 13791 consensus_queue.cc:1035] T 5f0bde561a284eb6abd6bf628ba2aace P d6b545ca970645febef66643334ce4ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "24be679363434337b186be2bbded83e9" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 36277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:47:00.911427 13798 mvcc.cc:205] Tried to move back new op lower bound from 7334237064775581696 to 7334237064640344064. Current Snapshot: MvccSnapshot[applied={T|T < 7334237064775581696}]
I20260928 08:47:00.923957 13800 mvcc.cc:205] Tried to move back new op lower bound from 7334237064775581696 to 7334237064640344064. Current Snapshot: MvccSnapshot[applied={T|T < 7334237064775581696}]
I20260928 08:47:00.957515 13281 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:47:00.957814 13281 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:47:00.962316 13281 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e6f3e4dd306497581f87a1072af880a: Sending DeleteTablet for 3 replicas of tablet 5f0bde561a284eb6abd6bf628ba2aace
I20260928 08:47:00.963708 13558 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f0bde561a284eb6abd6bf628ba2aace with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:47:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51930
I20260928 08:47:00.963765 13692 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f0bde561a284eb6abd6bf628ba2aace with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:47:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:34248
I20260928 08:47:00.964102 13413 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f0bde561a284eb6abd6bf628ba2aace with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:47:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:58162
I20260928 08:47:00.964332 13557 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f0bde561a284eb6abd6bf628ba2aace with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:47:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51930
I20260928 08:47:00.966020 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 13318
I20260928 08:47:00.983197 13807 tablet_replica.cc:330] T 5f0bde561a284eb6abd6bf628ba2aace P 24be679363434337b186be2bbded83e9: stopping tablet replica
I20260928 08:47:00.983886 13807 raft_consensus.cc:2229] T 5f0bde561a284eb6abd6bf628ba2aace P 24be679363434337b186be2bbded83e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:47:00.984668 13807 raft_consensus.cc:2258] T 5f0bde561a284eb6abd6bf628ba2aace P 24be679363434337b186be2bbded83e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:47:00.987461 13807 ts_tablet_manager.cc:1905] T 5f0bde561a284eb6abd6bf628ba2aace P 24be679363434337b186be2bbded83e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:47:00.987666 13809 tablet_replica.cc:330] T 5f0bde561a284eb6abd6bf628ba2aace P 5acbab1c85d348bca5d92cee3682e166: stopping tablet replica
I20260928 08:47:00.988293 13809 raft_consensus.cc:2229] T 5f0bde561a284eb6abd6bf628ba2aace P 5acbab1c85d348bca5d92cee3682e166 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:47:00.988724 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:00.967420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58174 (local address 127.0.97.1:38533)
0928 08:47:00.967638 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:00.967646 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:00.967667 (+ 21us) server_negotiation.cc:407] Connection header received
0928 08:47:00.967738 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:00.967746 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:00.968645 (+ 899us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:00.968761 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:00.971083 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:00.972627 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:00.975029 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:00.975701 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:00.975857 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:00.979968 (+ 4111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:00.979992 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:00.980018 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:00.980072 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:00.984516 (+ 4444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:00.985450 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:00.985459 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:00.985463 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:00.985627 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:00.986101 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:00.986105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:00.986108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:00.986547 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:00.986835 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:00.988102 (+ 1267us) server_negotiation.cc:299] Negotiation successful
0928 08:47:00.988329 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 08:47:00.989687 13473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:47:00.993664 13808 tablet_replica.cc:330] T 5f0bde561a284eb6abd6bf628ba2aace P d6b545ca970645febef66643334ce4ae: stopping tablet replica
I20260928 08:47:00.994112 13808 raft_consensus.cc:2229] T 5f0bde561a284eb6abd6bf628ba2aace P d6b545ca970645febef66643334ce4ae [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:47:00.994796 13808 raft_consensus.cc:2258] T 5f0bde561a284eb6abd6bf628ba2aace P d6b545ca970645febef66643334ce4ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:47:00.997573 13808 ts_tablet_manager.cc:1905] T 5f0bde561a284eb6abd6bf628ba2aace P d6b545ca970645febef66643334ce4ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:47:00.999511 13807 ts_tablet_manager.cc:1918] T 5f0bde561a284eb6abd6bf628ba2aace P 24be679363434337b186be2bbded83e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:47:00.999764 13807 log.cc:1196] T 5f0bde561a284eb6abd6bf628ba2aace P 24be679363434337b186be2bbded83e9: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-1/wal/wals/5f0bde561a284eb6abd6bf628ba2aace
I20260928 08:47:01.000685 13807 ts_tablet_manager.cc:1939] T 5f0bde561a284eb6abd6bf628ba2aace P 24be679363434337b186be2bbded83e9: Deleting consensus metadata
I20260928 08:47:01.008423 13808 ts_tablet_manager.cc:1918] T 5f0bde561a284eb6abd6bf628ba2aace P d6b545ca970645febef66643334ce4ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:47:01.008641 13808 log.cc:1196] T 5f0bde561a284eb6abd6bf628ba2aace P d6b545ca970645febef66643334ce4ae: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790585022442112-388-0/minicluster-data/ts-2/wal/wals/5f0bde561a284eb6abd6bf628ba2aace
I20260928 08:47:01.008931 13266 catalog_manager.cc:4841] TS 24be679363434337b186be2bbded83e9 (127.0.97.2:36277): tablet 5f0bde561a284eb6abd6bf628ba2aace (table test-table [id=0a422b3e2b1043eda593909484088c78]) successfully deleted
I20260928 08:47:01.009255 13808 ts_tablet_manager.cc:1939] T 5f0bde561a284eb6abd6bf628ba2aace P d6b545ca970645febef66643334ce4ae: Deleting consensus metadata
W20260928 08:47:01.010262 13266 catalog_manager.cc:4808] TS 24be679363434337b186be2bbded83e9 (127.0.97.2:36277): delete failed for tablet 5f0bde561a284eb6abd6bf628ba2aace because the tablet was not found. No further retry: Not found: Tablet not found: 5f0bde561a284eb6abd6bf628ba2aace
I20260928 08:47:01.010985 13266 catalog_manager.cc:4841] TS d6b545ca970645febef66643334ce4ae (127.0.97.3:38361): tablet 5f0bde561a284eb6abd6bf628ba2aace (table test-table [id=0a422b3e2b1043eda593909484088c78]) successfully deleted
I20260928 08:47:01.100914 13809 raft_consensus.cc:2258] T 5f0bde561a284eb6abd6bf628ba2aace P 5acbab1c85d348bca5d92cee3682e166 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:47:01.101563 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 13318
W20260928 08:47:01.119150 13265 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:38533 (error 108)
I20260928 08:47:01.119403 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 13482
I20260928 08:47:01.121472 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:01.121001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38533 (local address 127.0.0.1:58188)
0928 08:47:01.121192 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:01.121313 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 08:47:01.121891 13265 catalog_manager.cc:4584] TS 5acbab1c85d348bca5d92cee3682e166 (127.0.97.1:38533): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5f0bde561a284eb6abd6bf628ba2aace: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38533: connect: Connection refused (error 111)
I20260928 08:47:01.134044 13784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:01.120238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51932 (local address 127.0.97.2:36277)
0928 08:47:01.120428 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:01.120435 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:01.120595 (+ 160us) server_negotiation.cc:407] Connection header received
0928 08:47:01.120824 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:01.120829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:01.120896 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:01.121026 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:01.122490 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:01.123354 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:01.124847 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:01.125265 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:01.125410 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:01.128020 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:01.128037 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:01.128053 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:01.128090 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:01.131038 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:01.131756 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:01.131764 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:01.131766 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:01.132084 (+ 318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:01.132476 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:01.132483 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:01.132491 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:01.132834 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:01.133170 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:01.133438 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 08:47:01.133660 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 08:47:01.134804 13618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:47:01.175798 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:01.175158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38533 (local address 127.0.0.1:58200)
0928 08:47:01.175452 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:01.175585 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260928 08:47:01.177330 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:01.176858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38533 (local address 127.0.0.1:58202)
0928 08:47:01.177089 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:01.177178 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260928 08:47:01.238226 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:01.237649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38533 (local address 127.0.0.1:58204)
0928 08:47:01.237931 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:01.238044 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260928 08:47:01.239508 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:01.239048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38533 (local address 127.0.0.1:58208)
0928 08:47:01.239255 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:01.239363 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 08:47:01.242609 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 13482
I20260928 08:47:01.259404 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 13627
I20260928 08:47:01.276019 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:01.260193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34262 (local address 127.0.97.3:38361)
0928 08:47:01.260362 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:01.260366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:01.260447 (+ 81us) server_negotiation.cc:407] Connection header received
0928 08:47:01.260639 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:01.260643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:01.260707 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:01.260792 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:01.262655 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:01.263480 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:01.265318 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:01.265734 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:01.265831 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:01.269274 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:01.269287 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:01.269307 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:01.269347 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:01.272938 (+ 3591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:01.273772 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:01.273779 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:01.273782 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:01.273958 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:01.274386 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:01.274390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:01.274392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:01.274774 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:01.275032 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:01.275418 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 08:47:01.275594 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 08:47:01.276763 13752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:47:01.320003 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:01.319367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38533 (local address 127.0.0.1:58218)
0928 08:47:01.319608 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:01.319720 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260928 08:47:01.321174 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:01.320694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38533 (local address 127.0.0.1:58228)
0928 08:47:01.320912 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:01.321040 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:38533: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260928 08:47:01.400213 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 13627
I20260928 08:47:01.422554 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 13248
I20260928 08:47:01.423768 13311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:47:01.536203 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 13248
I20260928 08:47:01.561385 388 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 08:47:01.896886 388 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 08:47:01.897096 388 mini_ranger.cc:66] Stopping Ranger...
I20260928 08:47:02.242897 388 mini_ranger.cc:68] Stopped Ranger
2026-09-28 08:47:02.243 UTC [12300] LOG: received smart shutdown request
2026-09-28 08:47:02.247 UTC [12300] LOG: background worker "logical replication launcher" (PID 12309) exited with exit code 1
2026-09-28 08:47:02.247 UTC [12304] LOG: shutting down
2026-09-28 08:47:02.423 UTC [12300] LOG: database system is shut down
2026-09-28T08:47:02Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53496 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:47:02 dist-test-slave-n6xs krb5kdc[13823](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:47:02 dist-test-slave-n6xs krb5kdc[13823](info): set up 2 sockets
Sep 28 08:47:02 dist-test-slave-n6xs krb5kdc[13823](info): commencing operation
krb5kdc: starting...
W20260928 08:47:03.656201 388 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:47:03 dist-test-slave-n6xs krb5kdc[13823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:47:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:47:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:47:03.892989 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34491
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34491,127.0.97.61:45211,127.0.97.60:40857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:47:04.228484 13836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:47:04.228773 13836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:47:04.228932 13836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:47:04.236230 13836 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:47:04.236341 13836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:47:04.236430 13836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:47:04.236536 13836 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:47:04.236620 13836 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:47:04.246632 13836 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34491,127.0.97.61:45211,127.0.97.60:40857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:47:04.248106 13836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:47:04.249482 13836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:47:04.261479 13841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:04.261479 13842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:04.261520 13844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:47:04.262362 13836 server_base.cc:1034] running on GCE node
I20260928 08:47:04.263088 13836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:47:04.264526 13836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:47:04.265815 13836 hybrid_clock.cc:648] HybridClock initialized: now 1790585224265749 us; error 31 us; skew 500 ppm
I20260928 08:47:04.266233 13836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:47:04 dist-test-slave-n6xs krb5kdc[13823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:47:04.274467 13836 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:47:04.276202 13836 webserver.cc:460] Webserver started at http://127.0.97.62:36205/ using document root <none> and password file <none>
I20260928 08:47:04.276928 13836 fs_manager.cc:359] Metadata directory not provided
I20260928 08:47:04.277086 13836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:47:04.277433 13836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:47:04.280521 13836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "c1e42147924247afac06c81128a3143c"
format_stamp: "Formatted at 2026-09-28 08:47:04 on dist-test-slave-n6xs"
server_key: "c7280fb44c09884ce8c8975d21662cf1"
server_key_iv: "36f8c3914d565c505d7585972e0a8d65"
server_key_version: "encryptionkey@0"
I20260928 08:47:04.281307 13836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "c1e42147924247afac06c81128a3143c"
format_stamp: "Formatted at 2026-09-28 08:47:04 on dist-test-slave-n6xs"
server_key: "c7280fb44c09884ce8c8975d21662cf1"
server_key_iv: "36f8c3914d565c505d7585972e0a8d65"
server_key_version: "encryptionkey@0"
I20260928 08:47:04.286545 13836 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:47:04.289999 13851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:04.291416 13836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:47:04.291601 13836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "c1e42147924247afac06c81128a3143c"
format_stamp: "Formatted at 2026-09-28 08:47:04 on dist-test-slave-n6xs"
server_key: "c7280fb44c09884ce8c8975d21662cf1"
server_key_iv: "36f8c3914d565c505d7585972e0a8d65"
server_key_version: "encryptionkey@0"
I20260928 08:47:04.291893 13836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:47:04.307329 13836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:47:04.311524 13836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:47:04.312000 13836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:47:04.331763 13836 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34491
I20260928 08:47:04.331784 13902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34491 every 8 connection(s)
I20260928 08:47:04.333174 13836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:47:04.335551 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 13836
I20260928 08:47:04.335785 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:47:04.336623 388 external_mini_cluster.cc:1428] Setting key ed02259e6623a266c2e2bd770b4c06db
I20260928 08:47:04.337597 13903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:47:04.346920 13903 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34491 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 28 08:47:04 dist-test-slave-n6xs krb5kdc[13823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:47:04.370743 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:04.351710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50624 (local address 127.0.97.62:34491)
0928 08:47:04.352444 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:04.352452 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:04.352499 (+ 47us) server_negotiation.cc:407] Connection header received
0928 08:47:04.353668 (+ 1169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:04.353673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:04.353917 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:04.354125 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:04.355376 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:04.356665 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:04.358067 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:04.358660 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:04.358864 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:04.362339 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:04.362353 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:04.362372 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:04.362416 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:04.366416 (+ 4000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:04.367117 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:04.367128 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:04.367134 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:04.367320 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:04.367706 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:04.367713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:04.367719 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:04.368482 (+ 763us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:04.368689 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:04.369419 (+ 730us) server_negotiation.cc:299] Negotiation successful
0928 08:47:04.369679 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":324,"threads_started":1}
I20260928 08:47:04.370770 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:04.350289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34491 (local address 127.0.0.1:50624)
0928 08:47:04.351147 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:04.351185 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:04.353274 (+ 2089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:04.354147 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:04.354154 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:04.354851 (+ 697us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:47:04.355181 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:04.355198 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:04.356820 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:04.356826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:04.357925 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:04.357937 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:04.358891 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:04.358898 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:04.359099 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:04.359751 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:47:04.359772 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:47:04.362085 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:47:04.366564 (+ 4479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:04.366574 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:04.366583 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:04.366980 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:04.367430 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:04.367433 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:04.367435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:04.367585 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:04.368671 (+ 1086us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:47:04.368682 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:47:04.368972 (+ 290us) client_negotiation.cc:769] Sending connection context
0928 08:47:04.369288 (+ 316us) client_negotiation.cc:240] Negotiation successful
0928 08:47:04.369937 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":64,"thread_start_us":385,"threads_started":1}
I20260928 08:47:04.374217 13903 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45211 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:47:04.375308 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:04.374801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45211 (local address 127.0.0.1:33848)
0928 08:47:04.375016 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:04.375151 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260928 08:47:04.375568 13853 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45211: connect: Connection refused (error 111)
I20260928 08:47:04.377321 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:04.376846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45211 (local address 127.0.0.1:33864)
0928 08:47:04.377036 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:04.377141 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 08:47:04.377547 13903 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45211: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45211: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260928 08:47:04.391438 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:45211
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34491,127.0.97.61:45211,127.0.97.60:40857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:47:04.426934 13903 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45211 } attempt: 1
I20260928 08:47:04.428244 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:04.427686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45211 (local address 127.0.0.1:33874)
0928 08:47:04.427949 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:04.428083 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 08:47:04.429283 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:04.428911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45211 (local address 127.0.0.1:33884)
0928 08:47:04.429055 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:04.429133 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 08:47:04.429484 13903 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45211: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45211: connect: Connection refused (error 111)
I20260928 08:47:04.497735 13903 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45211 } attempt: 2
I20260928 08:47:04.498967 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:04.498462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45211 (local address 127.0.0.1:33892)
0928 08:47:04.498726 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:04.498824 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260928 08:47:04.500097 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:04.499694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45211 (local address 127.0.0.1:33900)
0928 08:47:04.499907 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:04.499976 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 08:47:04.500272 13903 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45211: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45211: connect: Connection refused (error 111)
I20260928 08:47:04.591572 13903 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45211 } attempt: 3
I20260928 08:47:04.592960 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:04.592261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45211 (local address 127.0.0.1:33902)
0928 08:47:04.592586 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:04.592740 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260928 08:47:04.594228 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:04.593730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45211 (local address 127.0.0.1:33908)
0928 08:47:04.593916 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:04.594031 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 08:47:04.594478 13903 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45211: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45211: connect: Connection refused (error 111)
W20260928 08:47:04.717804 13910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:47:04.718096 13910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:47:04.718186 13910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:47:04.725488 13910 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:47:04.725596 13910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:47:04.725643 13910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:47:04.725701 13910 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:47:04.725744 13910 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:47:04.736282 13910 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34491,127.0.97.61:45211,127.0.97.60:40857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:45211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:47:04.737737 13910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:47:04.737743 13903 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45211 } attempt: 4
I20260928 08:47:04.739053 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:04.738444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45211 (local address 127.0.0.1:33922)
0928 08:47:04.738762 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:04.738884 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260928 08:47:04.739424 13910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 08:47:04.740224 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:04.739777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45211 (local address 127.0.0.1:33938)
0928 08:47:04.739974 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:04.740077 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260928 08:47:04.740477 13903 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:45211: Network error: Client connection negotiation failed: client connection to 127.0.97.61:45211: connect: Connection refused (error 111)
W20260928 08:47:04.751542 13915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:04.751564 13916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:04.751686 13918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:47:04.752857 13910 server_base.cc:1034] running on GCE node
I20260928 08:47:04.753716 13910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:47:04.755174 13910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:47:04.756407 13910 hybrid_clock.cc:648] HybridClock initialized: now 1790585224756367 us; error 66 us; skew 500 ppm
I20260928 08:47:04.756748 13910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:47:04 dist-test-slave-n6xs krb5kdc[13823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:47:04.760115 13910 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260928 08:47:04.761606 13910 webserver.cc:460] Webserver started at http://127.0.97.61:33151/ using document root <none> and password file <none>
I20260928 08:47:04.762302 13910 fs_manager.cc:359] Metadata directory not provided
I20260928 08:47:04.762411 13910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:47:04.762669 13910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:47:04.765844 13910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-1/data/instance:
uuid: "a40e6463cb8d459ea1b5717555390000"
format_stamp: "Formatted at 2026-09-28 08:47:04 on dist-test-slave-n6xs"
server_key: "0844f081fc0b6f33d5dfd544f6b6e2fa"
server_key_iv: "720a83497d17dc163fbf9d5631822132"
server_key_version: "encryptionkey@0"
I20260928 08:47:04.766841 13910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-1/wal/instance:
uuid: "a40e6463cb8d459ea1b5717555390000"
format_stamp: "Formatted at 2026-09-28 08:47:04 on dist-test-slave-n6xs"
server_key: "0844f081fc0b6f33d5dfd544f6b6e2fa"
server_key_iv: "720a83497d17dc163fbf9d5631822132"
server_key_version: "encryptionkey@0"
I20260928 08:47:04.772188 13910 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 08:47:04.775512 13925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:04.776940 13910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:47:04.777081 13910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-1/wal
uuid: "a40e6463cb8d459ea1b5717555390000"
format_stamp: "Formatted at 2026-09-28 08:47:04 on dist-test-slave-n6xs"
server_key: "0844f081fc0b6f33d5dfd544f6b6e2fa"
server_key_iv: "720a83497d17dc163fbf9d5631822132"
server_key_version: "encryptionkey@0"
I20260928 08:47:04.777253 13910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:47:04.794499 13910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:47:04.798825 13910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:47:04.799053 13910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:47:04.818025 13910 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:45211
I20260928 08:47:04.818056 13976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:45211 every 8 connection(s)
I20260928 08:47:04.819458 13910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-1/data/info.pb
I20260928 08:47:04.822743 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 13910
I20260928 08:47:04.822945 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-1/wal/instance
I20260928 08:47:04.823341 13977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:47:04.823416 388 external_mini_cluster.cc:1428] Setting key 226edaabd6214519fff5ff6edc9cc8d0
I20260928 08:47:04.829892 13977 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34491 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 28 08:47:04 dist-test-slave-n6xs krb5kdc[13823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:47:04.854228 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:04.833155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50632 (local address 127.0.97.62:34491)
0928 08:47:04.833389 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:04.833398 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:04.834594 (+ 1196us) server_negotiation.cc:407] Connection header received
0928 08:47:04.836555 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:04.836563 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:04.836658 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:04.836782 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:04.838223 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:04.839395 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:04.841090 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:04.841885 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:04.842036 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:04.847112 (+ 5076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:04.847122 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:04.847131 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:04.847180 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:04.850997 (+ 3817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:04.851768 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:04.851776 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:04.851783 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:04.851958 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:04.852383 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:04.852390 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:04.852396 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:04.852799 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:04.852956 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:04.853809 (+ 853us) server_negotiation.cc:299] Negotiation successful
0928 08:47:04.853986 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 08:47:04.855763 13980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:04.833592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34491 (local address 127.0.0.1:50632)
0928 08:47:04.834415 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:04.834460 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:04.836283 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:04.836981 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:04.836989 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:04.837509 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:47:04.838048 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:04.838064 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:04.839580 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:04.839584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:04.840947 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:04.840959 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:04.842089 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:04.842094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:04.842281 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:04.842991 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:47:04.843012 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:47:04.846868 (+ 3856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:47:04.851164 (+ 4296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:04.851171 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:04.851184 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:04.851634 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:04.852077 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:04.852080 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:04.852082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:04.852257 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:04.852917 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:47:04.852923 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:47:04.853381 (+ 458us) client_negotiation.cc:769] Sending connection context
0928 08:47:04.853711 (+ 330us) client_negotiation.cc:240] Negotiation successful
0928 08:47:04.854272 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":385,"threads_started":1}
I20260928 08:47:04.857695 13977 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45211 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:47:04 dist-test-slave-n6xs krb5kdc[13823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 08:47:04.871850 13980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:04.858333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45211 (local address 127.0.0.1:33954)
0928 08:47:04.858543 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:04.858558 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:04.858636 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:04.859643 (+ 1007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:04.859647 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:04.860106 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:47:04.860247 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:04.860254 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:04.861570 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:04.861577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:04.862406 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:04.862416 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:04.863210 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:04.863216 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:04.863434 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:04.864067 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:47:04.864083 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:47:04.866004 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:47:04.869274 (+ 3270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:04.869279 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:04.869283 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:04.869633 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:04.870070 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:04.870078 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:04.870082 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:04.870218 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:04.870795 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:47:04.870804 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:47:04.870998 (+ 194us) client_negotiation.cc:769] Sending connection context
0928 08:47:04.871133 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 08:47:04.871415 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 08:47:04.871934 13982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:04.858659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33954 (local address 127.0.97.61:45211)
0928 08:47:04.859255 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:04.859260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:04.859284 (+ 24us) server_negotiation.cc:407] Connection header received
0928 08:47:04.859325 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:04.859333 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:04.859507 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:04.859655 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:04.860398 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:04.861423 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:04.862532 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:04.863071 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:04.863202 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:04.866205 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:04.866215 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:04.866228 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:04.866260 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:04.869138 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:04.869772 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:04.869778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:04.869780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:04.869952 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:04.870317 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:04.870323 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:04.870328 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:04.870679 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:04.870903 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:04.871281 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 08:47:04.871523 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":463,"thread_start_us":332,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260928 08:47:04.874372 13977 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40857 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:47:04.875573 13980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:04.875032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40857 (local address 127.0.0.1:56186)
0928 08:47:04.875215 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:04.875367 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 08:47:04.875641 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:40857
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34491,127.0.97.61:45211,127.0.97.60:40857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:47:04.875850 13929 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40857: connect: Connection refused (error 111)
I20260928 08:47:04.877512 13980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:04.877092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40857 (local address 127.0.0.1:56188)
0928 08:47:04.877264 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:04.877371 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 08:47:04.877732 13977 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40857: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40857: connect: Connection refused (error 111)
I20260928 08:47:04.927079 13977 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40857 } attempt: 1
I20260928 08:47:04.928450 13980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:04.927931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40857 (local address 127.0.0.1:56200)
0928 08:47:04.928180 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:04.928287 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260928 08:47:04.929531 13980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:04.929166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40857 (local address 127.0.0.1:56204)
0928 08:47:04.929351 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:04.929431 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 08:47:04.929733 13977 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40857: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40857: connect: Connection refused (error 111)
I20260928 08:47:04.997948 13977 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40857 } attempt: 2
I20260928 08:47:04.999110 13980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:04.998657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40857 (local address 127.0.0.1:56206)
0928 08:47:04.998850 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:04.998961 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 08:47:05.000413 13980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:04.999967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40857 (local address 127.0.0.1:56210)
0928 08:47:05.000152 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:05.000229 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 08:47:05.000634 13977 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40857: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40857: connect: Connection refused (error 111)
I20260928 08:47:05.039757 13903 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45211 } attempt: 5
Sep 28 08:47:05 dist-test-slave-n6xs krb5kdc[13823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 08:47:05.054227 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:05.040579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45211 (local address 127.0.0.1:33956)
0928 08:47:05.040847 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:05.040870 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:05.041033 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:05.041497 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:05.041503 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:05.042014 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:47:05.042205 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:05.042214 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.043292 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.043300 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:05.044275 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:05.044287 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.045125 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.045131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:05.045345 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:05.046035 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:47:05.046054 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:47:05.048288 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:47:05.051529 (+ 3241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:05.051536 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:05.051540 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:05.052038 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:05.052441 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:05.052447 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:05.052451 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:05.052612 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:05.053186 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:47:05.053192 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:47:05.053356 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 08:47:05.053517 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 08:47:05.053844 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260928 08:47:05.054235 13982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:05.040716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33956 (local address 127.0.97.61:45211)
0928 08:47:05.040958 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:05.040964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:05.040988 (+ 24us) server_negotiation.cc:407] Connection header received
0928 08:47:05.041195 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:05.041201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:05.041279 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:05.041457 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:05.042390 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.043134 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.044430 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.044959 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.045115 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:05.048499 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:05.048505 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:05.048509 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:05.048544 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:05.051371 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:05.052164 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:05.052172 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:05.052176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:05.052318 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:05.052741 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:05.052751 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:05.052754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:05.053056 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:05.053228 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:05.053644 (+ 416us) server_negotiation.cc:299] Negotiation successful
0928 08:47:05.053867 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 08:47:05.055709 13903 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40857 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:47:05.056938 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:05.056548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40857 (local address 127.0.0.1:56216)
0928 08:47:05.056715 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:05.056802 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 08:47:05.057842 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:05.057535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40857 (local address 127.0.0.1:56226)
0928 08:47:05.057669 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:05.057752 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 08:47:05.058072 13903 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40857: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40857: connect: Connection refused (error 111)
I20260928 08:47:05.091936 13977 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40857 } attempt: 3
I20260928 08:47:05.093202 13980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:05.092643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40857 (local address 127.0.0.1:56236)
0928 08:47:05.092890 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:05.093008 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 08:47:05.094504 13980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:05.094028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40857 (local address 127.0.0.1:56242)
0928 08:47:05.094267 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:05.094383 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
W20260928 08:47:05.094818 13977 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40857: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40857: connect: Connection refused (error 111)
I20260928 08:47:05.109279 13903 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40857 } attempt: 1
I20260928 08:47:05.110512 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:05.109936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40857 (local address 127.0.0.1:56248)
0928 08:47:05.110178 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:05.110299 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 08:47:05.111672 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:05.111211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40857 (local address 127.0.0.1:56254)
0928 08:47:05.111396 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:05.111519 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260928 08:47:05.111951 13903 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40857: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40857: connect: Connection refused (error 111)
I20260928 08:47:05.180248 13903 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40857 } attempt: 2
I20260928 08:47:05.181594 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:05.181003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40857 (local address 127.0.0.1:56268)
0928 08:47:05.181322 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:05.181431 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260928 08:47:05.182657 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:05.182301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40857 (local address 127.0.0.1:56284)
0928 08:47:05.182471 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:05.182547 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260928 08:47:05.182943 13903 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40857: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40857: connect: Connection refused (error 111)
W20260928 08:47:05.203616 13984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:47:05.203969 13984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:47:05.204149 13984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:47:05.211319 13984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:47:05.211412 13984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:47:05.211522 13984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:47:05.211623 13984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:47:05.211707 13984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:47:05.221566 13984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34491,127.0.97.61:45211,127.0.97.60:40857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:40857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:47:05.222870 13984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:47:05.224207 13984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:47:05.236097 13989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:05.236060 13992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:05.236176 13990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:47:05.237007 13984 server_base.cc:1034] running on GCE node
I20260928 08:47:05.237748 13984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:47:05.238013 13977 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40857 } attempt: 4
I20260928 08:47:05.238996 13980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:05.238552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40857 (local address 127.0.0.1:56298)
0928 08:47:05.238763 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:05.238855 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 08:47:05.239200 13984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:47:05.240108 13980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:05.239793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40857 (local address 127.0.0.1:56302)
0928 08:47:05.239950 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:05.240012 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260928 08:47:05.240301 13977 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40857: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40857: connect: Connection refused (error 111)
I20260928 08:47:05.240417 13984 hybrid_clock.cc:648] HybridClock initialized: now 1790585225240339 us; error 47 us; skew 500 ppm
I20260928 08:47:05.240768 13984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:47:05 dist-test-slave-n6xs krb5kdc[13823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:47:05.244059 13984 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260928 08:47:05.245540 13984 webserver.cc:460] Webserver started at http://127.0.97.60:39993/ using document root <none> and password file <none>
I20260928 08:47:05.246194 13984 fs_manager.cc:359] Metadata directory not provided
I20260928 08:47:05.246366 13984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:47:05.246659 13984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:47:05.249627 13984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-2/data/instance:
uuid: "3ea965677eaa450487b3fa6edb175f3b"
format_stamp: "Formatted at 2026-09-28 08:47:05 on dist-test-slave-n6xs"
server_key: "a701b101de7447ae15f75913c0eb8849"
server_key_iv: "6a6eb8b6bff6e2ee3a6d7c8d9dc1a4ad"
server_key_version: "encryptionkey@0"
I20260928 08:47:05.250443 13984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-2/wal/instance:
uuid: "3ea965677eaa450487b3fa6edb175f3b"
format_stamp: "Formatted at 2026-09-28 08:47:05 on dist-test-slave-n6xs"
server_key: "a701b101de7447ae15f75913c0eb8849"
server_key_iv: "6a6eb8b6bff6e2ee3a6d7c8d9dc1a4ad"
server_key_version: "encryptionkey@0"
I20260928 08:47:05.255343 13984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 08:47:05.259264 13999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:05.260649 13984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:47:05.260821 13984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-2/wal
uuid: "3ea965677eaa450487b3fa6edb175f3b"
format_stamp: "Formatted at 2026-09-28 08:47:05 on dist-test-slave-n6xs"
server_key: "a701b101de7447ae15f75913c0eb8849"
server_key_iv: "6a6eb8b6bff6e2ee3a6d7c8d9dc1a4ad"
server_key_version: "encryptionkey@0"
I20260928 08:47:05.261026 13984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:47:05.282143 13984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:47:05.288120 13984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:47:05.288352 13984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:47:05.289183 13903 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40857 } attempt: 3
I20260928 08:47:05.290402 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:05.289836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40857 (local address 127.0.0.1:56310)
0928 08:47:05.290111 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:05.290208 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 08:47:05.291587 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:05.291101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40857 (local address 127.0.0.1:56312)
0928 08:47:05.291344 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:05.291444 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260928 08:47:05.291836 13903 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:40857: Network error: Client connection negotiation failed: client connection to 127.0.97.60:40857: connect: Connection refused (error 111)
I20260928 08:47:05.307386 13984 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:40857
I20260928 08:47:05.307503 14050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:40857 every 8 connection(s)
I20260928 08:47:05.308840 13984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-2/data/info.pb
I20260928 08:47:05.312582 14051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:47:05.317214 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 13984
I20260928 08:47:05.317412 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790585022442112-388-0/minicluster-data/master-2/wal/instance
I20260928 08:47:05.317840 388 external_mini_cluster.cc:1428] Setting key 8d2b9b2bf45e6d843fdd7339eac1a263
I20260928 08:47:05.319379 14051 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34491 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:47:05 dist-test-slave-n6xs krb5kdc[13823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 28 08:47:05 dist-test-slave-n6xs krb5kdc[13823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260928 08:47:05.338624 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:05.322342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50640 (local address 127.0.97.62:34491)
0928 08:47:05.322566 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:05.322574 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:05.323362 (+ 788us) server_negotiation.cc:407] Connection header received
0928 08:47:05.323605 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:05.323611 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:05.323682 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:05.323788 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:05.325029 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.325707 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.327180 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.327629 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.327761 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:05.331730 (+ 3969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:05.331738 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:05.331744 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:05.331850 (+ 106us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:05.335114 (+ 3264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:05.335922 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:05.335929 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:05.335933 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:05.336091 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:05.336561 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:05.336566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:05.336569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:05.336846 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:05.337025 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:05.337989 (+ 964us) server_negotiation.cc:299] Negotiation successful
0928 08:47:05.338230 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 08:47:05.339646 14055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:05.322350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:34491 (local address 127.0.0.1:50640)
0928 08:47:05.323195 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:05.323241 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:05.323400 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:05.323873 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:05.323879 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:05.324496 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:47:05.324860 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:05.324877 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.325872 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.325876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:05.327030 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:05.327041 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.327911 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.327920 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:05.328129 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:05.328816 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:47:05.328838 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:47:05.331510 (+ 2672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:47:05.335244 (+ 3734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:05.335257 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:05.335269 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:05.335757 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:05.336208 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:05.336215 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:05.336219 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:05.336432 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:05.337103 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:47:05.337111 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:47:05.337532 (+ 421us) client_negotiation.cc:769] Sending connection context
0928 08:47:05.337861 (+ 329us) client_negotiation.cc:240] Negotiation successful
0928 08:47:05.338354 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":467,"threads_started":1}
I20260928 08:47:05.340549 14054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:05.319657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56326 (local address 127.0.97.60:40857)
0928 08:47:05.320575 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:05.320596 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:05.320648 (+ 52us) server_negotiation.cc:407] Connection header received
0928 08:47:05.322797 (+ 2149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:05.322821 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:05.323197 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:05.323570 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:05.325147 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.326326 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.327713 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.328333 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.328481 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:05.333251 (+ 4770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:05.333261 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:05.333277 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:05.333315 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:05.336805 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:05.337619 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:05.337628 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:05.337630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:05.337856 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:05.338333 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:05.338338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:05.338342 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:05.338756 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:05.339055 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:05.339692 (+ 637us) server_negotiation.cc:299] Negotiation successful
0928 08:47:05.339962 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"spinlock_wait_cycles":19328,"thread_start_us":359,"threads_started":1}
I20260928 08:47:05.341591 14051 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45211 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:47:05 dist-test-slave-n6xs krb5kdc[13823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 28 08:47:05 dist-test-slave-n6xs krb5kdc[13823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:47:05.356348 14055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:05.342248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:45211 (local address 127.0.0.1:33960)
0928 08:47:05.342479 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:05.342499 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:05.342597 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:05.343083 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:05.343087 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:05.343535 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:47:05.343733 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:05.343743 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.345255 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.345259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:05.346597 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:05.346612 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.347373 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.347379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:05.347606 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:05.348248 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:47:05.348269 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:47:05.350393 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:47:05.353879 (+ 3486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:05.353886 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:05.353890 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:05.354246 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:05.354640 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:05.354646 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:05.354649 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:05.354799 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:05.355371 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:47:05.355376 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:47:05.355533 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 08:47:05.355673 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 08:47:05.355968 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 08:47:05.356371 13982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:05.342597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33960 (local address 127.0.97.61:45211)
0928 08:47:05.342799 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:05.342807 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:05.342826 (+ 19us) server_negotiation.cc:407] Connection header received
0928 08:47:05.342887 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:05.342891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:05.342949 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:05.343061 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:05.343932 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.345090 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.346777 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.347221 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.347357 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:05.350583 (+ 3226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:05.350591 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:05.350600 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:05.350633 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:05.353751 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:05.354381 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:05.354387 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:05.354390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:05.354529 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:05.354911 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:05.354918 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:05.354935 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:05.355253 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:05.355415 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:05.355760 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 08:47:05.356010 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 08:47:05.357332 14051 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40857 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:47:05.358661 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:05.343417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50652 (local address 127.0.97.62:34491)
0928 08:47:05.343576 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:05.343582 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:05.343623 (+ 41us) server_negotiation.cc:407] Connection header received
0928 08:47:05.343738 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:05.343743 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:05.343839 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:05.343941 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:05.345474 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.346555 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.348013 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.348608 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.348739 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:05.353004 (+ 4265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:05.353011 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:05.353015 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:05.353062 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:05.355935 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:05.356666 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:05.356670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:05.356673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:05.356808 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:05.357176 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:05.357182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:05.357185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:05.357541 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:05.357817 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:05.358113 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 08:47:05.358309 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 08:47:05 dist-test-slave-n6xs krb5kdc[13823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585225, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 28 08:47:05 dist-test-slave-n6xs krb5kdc[13823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 08:47:05.371775 14054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:05.358396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56330 (local address 127.0.97.60:40857)
0928 08:47:05.358717 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:05.358723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:05.358791 (+ 68us) server_negotiation.cc:407] Connection header received
0928 08:47:05.358955 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:05.358960 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:05.359018 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:05.359115 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:05.359862 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.360868 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.362044 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.362529 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.362619 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:05.365961 (+ 3342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:05.365968 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:05.365974 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:05.366014 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:05.369016 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:05.369668 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:05.369676 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:05.369682 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:05.369811 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:05.370226 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:05.370231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:05.370235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:05.370607 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:05.370761 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:05.371103 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 08:47:05.371472 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":164,"server-negotiator.queue_time_us":213}
I20260928 08:47:05.371783 14055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:05.358189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40857 (local address 127.0.0.1:56330)
0928 08:47:05.358413 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:05.358747 (+ 334us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:05.358863 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:05.359129 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:05.359133 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:05.359501 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:47:05.359663 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:05.359674 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.361019 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.361025 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:05.361904 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:05.361916 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.362665 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.362671 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:05.362867 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:05.363456 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:47:05.363472 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:47:05.365752 (+ 2280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:47:05.369189 (+ 3437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:05.369194 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:05.369198 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:05.369553 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:05.369968 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:05.369972 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:05.369974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:05.370115 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:05.370725 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:47:05.370730 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:47:05.370870 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 08:47:05.371015 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 08:47:05.371343 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":3}
I20260928 08:47:05.375923 13982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:05.360455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33976 (local address 127.0.97.61:45211)
0928 08:47:05.360610 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:05.360614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:05.360649 (+ 35us) server_negotiation.cc:407] Connection header received
0928 08:47:05.360800 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:05.360803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:05.360854 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:05.360928 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:05.362091 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.363011 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.364392 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.365058 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.365190 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:05.369966 (+ 4776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:05.369973 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:05.369978 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:05.370028 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:05.373206 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:05.373864 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:05.373870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:05.373876 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:05.374025 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:05.374474 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:05.374479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:05.374482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:05.374823 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:05.375102 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:05.375334 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 08:47:05.375559 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 08:47:05.378228 14051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ea965677eaa450487b3fa6edb175f3b: Bootstrap starting.
I20260928 08:47:05.381490 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:47:05.382117 14051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ea965677eaa450487b3fa6edb175f3b: Neither blocks nor log segments found. Creating new log.
I20260928 08:47:05.383415 14051 log.cc:824] T 00000000000000000000000000000000 P 3ea965677eaa450487b3fa6edb175f3b: Log is configured to *not* fsync() on all Append() calls
I20260928 08:47:05.386839 14051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ea965677eaa450487b3fa6edb175f3b: No bootstrap required, opened a new log
I20260928 08:47:05.398988 14051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ea965677eaa450487b3fa6edb175f3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e42147924247afac06c81128a3143c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34491 } } peers { permanent_uuid: "a40e6463cb8d459ea1b5717555390000" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45211 } } peers { permanent_uuid: "3ea965677eaa450487b3fa6edb175f3b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40857 } }
I20260928 08:47:05.399622 14051 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ea965677eaa450487b3fa6edb175f3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:47:05.399835 14051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ea965677eaa450487b3fa6edb175f3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ea965677eaa450487b3fa6edb175f3b, State: Initialized, Role: FOLLOWER
I20260928 08:47:05.400665 14051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ea965677eaa450487b3fa6edb175f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e42147924247afac06c81128a3143c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34491 } } peers { permanent_uuid: "a40e6463cb8d459ea1b5717555390000" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45211 } } peers { permanent_uuid: "3ea965677eaa450487b3fa6edb175f3b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40857 } }
I20260928 08:47:05.407508 13907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:05.387175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50660 (local address 127.0.97.62:34491)
0928 08:47:05.387339 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:05.387345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:05.388201 (+ 856us) server_negotiation.cc:407] Connection header received
0928 08:47:05.388522 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:05.388528 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:05.388628 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:05.388711 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:05.390923 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.392053 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.394206 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.394862 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.394960 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:05.399022 (+ 4062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:05.399028 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:05.399034 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:05.399075 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:05.403139 (+ 4064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:05.404285 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:05.404294 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:05.404315 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:05.404512 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:05.405091 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:05.405099 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:05.405104 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:05.405590 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:05.405890 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:05.406628 (+ 738us) server_negotiation.cc:299] Negotiation successful
0928 08:47:05.406959 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 08:47:05.409932 14064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ea965677eaa450487b3fa6edb175f3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e42147924247afac06c81128a3143c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34491 } } peers { permanent_uuid: "a40e6463cb8d459ea1b5717555390000" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45211 } } peers { permanent_uuid: "3ea965677eaa450487b3fa6edb175f3b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40857 } } }
I20260928 08:47:05.410463 14064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ea965677eaa450487b3fa6edb175f3b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:47:05.410625 14054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:05.387357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56332 (local address 127.0.97.60:40857)
0928 08:47:05.387577 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:05.387583 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:05.388543 (+ 960us) server_negotiation.cc:407] Connection header received
0928 08:47:05.388792 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:05.388800 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:05.388877 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:05.388969 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:05.391931 (+ 2962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.393415 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.396310 (+ 2895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.397339 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.397466 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:05.402063 (+ 4597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:05.402070 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:05.402074 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:05.402117 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:05.405689 (+ 3572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:05.406560 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:05.406567 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:05.406570 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:05.406735 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:05.407239 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:05.407250 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:05.407256 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:05.407728 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:05.408009 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:05.408657 (+ 648us) server_negotiation.cc:299] Negotiation successful
0928 08:47:05.408909 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260928 08:47:05.411525 14051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ea965677eaa450487b3fa6edb175f3b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:47:05.418946 13982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:05.388930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33980 (local address 127.0.97.61:45211)
0928 08:47:05.389105 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:05.389112 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:05.399736 (+ 10624us) server_negotiation.cc:407] Connection header received
0928 08:47:05.400221 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:05.400229 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:05.400324 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:05.400417 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:05.402298 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.403481 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.405554 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.406217 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.406323 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:05.409783 (+ 3460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:05.409789 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:05.409793 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:05.409832 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:05.414540 (+ 4708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:05.415677 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:05.415686 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:05.415692 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:05.415964 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:05.416717 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:05.416724 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:05.416729 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:05.417107 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:05.417387 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:05.418334 (+ 947us) server_negotiation.cc:299] Negotiation successful
0928 08:47:05.418588 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 08:47:05.426913 14075 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3ea965677eaa450487b3fa6edb175f3b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:47:05.427057 14075 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:47:05.422847 13984 master_runner.cc:186] Error getting master registration for 127.0.97.62:34491: OK, instance_id { permanent_uuid: "c1e42147924247afac06c81128a3143c" instance_seqno: 1790585224325974 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 08:47:05.431169 13984 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 08:47:05.469043 13903 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40857 } attempt: 4
Sep 28 08:47:05 dist-test-slave-n6xs krb5kdc[13823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260928 08:47:05.484481 13906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:05.469685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40857 (local address 127.0.0.1:56336)
0928 08:47:05.469923 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:05.469945 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:05.470132 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:05.471117 (+ 985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:05.471121 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:05.471675 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:47:05.471970 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:05.471981 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.473108 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.473114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:05.474040 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:05.474051 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.474834 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.474841 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:05.475041 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:05.475675 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:47:05.475691 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:47:05.478065 (+ 2374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:47:05.482021 (+ 3956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:05.482025 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:05.482028 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:05.482388 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:05.482789 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:05.482793 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:05.482795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:05.482927 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:05.483516 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:47:05.483519 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:47:05.483683 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 08:47:05.483853 (+ 170us) client_negotiation.cc:240] Negotiation successful
0928 08:47:05.484201 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 08:47:05.484584 14054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:05.469782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56336 (local address 127.0.97.60:40857)
0928 08:47:05.469984 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:05.469992 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:05.470027 (+ 35us) server_negotiation.cc:407] Connection header received
0928 08:47:05.470315 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:05.470320 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:05.470441 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:05.471722 (+ 1281us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:05.472118 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.472958 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.474185 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.474638 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.474801 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:05.478251 (+ 3450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:05.478256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:05.478259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:05.478288 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:05.481892 (+ 3604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:05.482495 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:05.482501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:05.482505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:05.482680 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:05.483025 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:05.483030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:05.483034 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:05.483396 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:05.483542 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:05.483973 (+ 431us) server_negotiation.cc:299] Negotiation successful
0928 08:47:05.484210 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 08:47:05.488858 13903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1e42147924247afac06c81128a3143c: Bootstrap starting.
I20260928 08:47:05.492659 13903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1e42147924247afac06c81128a3143c: Neither blocks nor log segments found. Creating new log.
I20260928 08:47:05.493717 13903 log.cc:824] T 00000000000000000000000000000000 P c1e42147924247afac06c81128a3143c: Log is configured to *not* fsync() on all Append() calls
I20260928 08:47:05.496524 13903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1e42147924247afac06c81128a3143c: No bootstrap required, opened a new log
I20260928 08:47:05.503504 13903 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1e42147924247afac06c81128a3143c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e42147924247afac06c81128a3143c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34491 } } peers { permanent_uuid: "a40e6463cb8d459ea1b5717555390000" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45211 } } peers { permanent_uuid: "3ea965677eaa450487b3fa6edb175f3b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40857 } }
I20260928 08:47:05.503924 13903 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1e42147924247afac06c81128a3143c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:47:05.504076 13903 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1e42147924247afac06c81128a3143c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1e42147924247afac06c81128a3143c, State: Initialized, Role: FOLLOWER
I20260928 08:47:05.504756 13903 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1e42147924247afac06c81128a3143c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e42147924247afac06c81128a3143c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34491 } } peers { permanent_uuid: "a40e6463cb8d459ea1b5717555390000" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45211 } } peers { permanent_uuid: "3ea965677eaa450487b3fa6edb175f3b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40857 } }
I20260928 08:47:05.506078 14078 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1e42147924247afac06c81128a3143c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e42147924247afac06c81128a3143c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34491 } } peers { permanent_uuid: "a40e6463cb8d459ea1b5717555390000" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45211 } } peers { permanent_uuid: "3ea965677eaa450487b3fa6edb175f3b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40857 } } }
I20260928 08:47:05.506486 14078 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1e42147924247afac06c81128a3143c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:47:05.507222 13903 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1e42147924247afac06c81128a3143c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:47:05.510949 14078 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1e42147924247afac06c81128a3143c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:47:05.511241 14078 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1e42147924247afac06c81128a3143c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e42147924247afac06c81128a3143c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34491 } } peers { permanent_uuid: "a40e6463cb8d459ea1b5717555390000" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45211 } } peers { permanent_uuid: "3ea965677eaa450487b3fa6edb175f3b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40857 } }
I20260928 08:47:05.513002 14078 leader_election.cc:288] T 00000000000000000000000000000000 P c1e42147924247afac06c81128a3143c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a40e6463cb8d459ea1b5717555390000 (127.0.97.61:45211), 3ea965677eaa450487b3fa6edb175f3b (127.0.97.60:40857)
I20260928 08:47:05.514063 14026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c1e42147924247afac06c81128a3143c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ea965677eaa450487b3fa6edb175f3b" is_pre_election: true
I20260928 08:47:05.514683 14026 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3ea965677eaa450487b3fa6edb175f3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1e42147924247afac06c81128a3143c in term 0.
I20260928 08:47:05.515362 13855 leader_election.cc:302] T 00000000000000000000000000000000 P c1e42147924247afac06c81128a3143c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ea965677eaa450487b3fa6edb175f3b, c1e42147924247afac06c81128a3143c; no voters:
I20260928 08:47:05.515776 14078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1e42147924247afac06c81128a3143c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:47:05.515936 14078 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1e42147924247afac06c81128a3143c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:47:05.516047 14078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1e42147924247afac06c81128a3143c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:47:05.521800 14078 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1e42147924247afac06c81128a3143c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e42147924247afac06c81128a3143c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34491 } } peers { permanent_uuid: "a40e6463cb8d459ea1b5717555390000" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45211 } } peers { permanent_uuid: "3ea965677eaa450487b3fa6edb175f3b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40857 } }
I20260928 08:47:05.522969 14026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c1e42147924247afac06c81128a3143c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ea965677eaa450487b3fa6edb175f3b"
I20260928 08:47:05.523182 14026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ea965677eaa450487b3fa6edb175f3b [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:47:05.526660 14026 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3ea965677eaa450487b3fa6edb175f3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1e42147924247afac06c81128a3143c in term 1.
I20260928 08:47:05.527055 14078 leader_election.cc:288] T 00000000000000000000000000000000 P c1e42147924247afac06c81128a3143c [CANDIDATE]: Term 1 election: Requested vote from peers a40e6463cb8d459ea1b5717555390000 (127.0.97.61:45211), 3ea965677eaa450487b3fa6edb175f3b (127.0.97.60:40857)
I20260928 08:47:05.527100 13855 leader_election.cc:302] T 00000000000000000000000000000000 P c1e42147924247afac06c81128a3143c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ea965677eaa450487b3fa6edb175f3b, c1e42147924247afac06c81128a3143c; no voters:
I20260928 08:47:05.529170 14078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1e42147924247afac06c81128a3143c [term 1 FOLLOWER]: Leader election won for term 1
W20260928 08:47:05.530473 14089 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c1e42147924247afac06c81128a3143c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:47:05.530706 14089 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:47:05.531332 14078 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1e42147924247afac06c81128a3143c [term 1 LEADER]: Becoming Leader. State: Replica: c1e42147924247afac06c81128a3143c, State: Running, Role: LEADER
I20260928 08:47:05.531953 14078 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1e42147924247afac06c81128a3143c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e42147924247afac06c81128a3143c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34491 } } peers { permanent_uuid: "a40e6463cb8d459ea1b5717555390000" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45211 } } peers { permanent_uuid: "3ea965677eaa450487b3fa6edb175f3b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40857 } }
I20260928 08:47:05.536918 14091 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1e42147924247afac06c81128a3143c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1e42147924247afac06c81128a3143c. Latest consensus state: current_term: 1 leader_uuid: "c1e42147924247afac06c81128a3143c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e42147924247afac06c81128a3143c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34491 } } peers { permanent_uuid: "a40e6463cb8d459ea1b5717555390000" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45211 } } peers { permanent_uuid: "3ea965677eaa450487b3fa6edb175f3b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40857 } } }
I20260928 08:47:05.538967 14091 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1e42147924247afac06c81128a3143c [sys.catalog]: This master's current role is: LEADER
I20260928 08:47:05.539507 13977 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40857 } attempt: 5
W20260928 08:47:05.540519 14005 tablet.cc:2369] T 00000000000000000000000000000000 P 3ea965677eaa450487b3fa6edb175f3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:47:05.518478 13836 master_runner.cc:186] Error getting master registration for 127.0.97.61:45211: OK, instance_id { permanent_uuid: "a40e6463cb8d459ea1b5717555390000" instance_seqno: 1790585224812691 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 08:47:05.546223 13836 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 08:47:05.547943 14093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 28 08:47:05 dist-test-slave-n6xs krb5kdc[13823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260928 08:47:05.550172 14093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:47:05.559562 13980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:05.540073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:40857 (local address 127.0.0.1:56340)
0928 08:47:05.540262 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:05.540287 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:05.540414 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:05.540920 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:05.540925 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:05.541471 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:47:05.541675 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:05.541686 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.543102 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.543108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:05.545035 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:05.545050 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.546058 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.546064 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:05.546262 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:05.546982 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:47:05.547005 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:47:05.549860 (+ 2855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:47:05.555904 (+ 6044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:05.555911 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:05.555915 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:05.556436 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:05.557208 (+ 772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:05.557213 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:05.557217 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:05.557394 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:05.558537 (+ 1143us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:47:05.558543 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:47:05.558725 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 08:47:05.558860 (+ 135us) client_negotiation.cc:240] Negotiation successful
0928 08:47:05.559171 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 08:47:05.562033 14054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:05.540147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56340 (local address 127.0.97.60:40857)
0928 08:47:05.540342 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:05.540348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:05.540370 (+ 22us) server_negotiation.cc:407] Connection header received
0928 08:47:05.540550 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:05.540555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:05.540635 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:05.540738 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:05.541845 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.542935 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.545203 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:05.545917 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:05.546055 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:05.550071 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:05.550078 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:05.550083 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:05.550121 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:05.555600 (+ 5479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:05.556695 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:05.556701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:05.556705 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:05.556981 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:05.557588 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:05.557593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:05.557597 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:05.558341 (+ 744us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:05.560387 (+ 2046us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:05.560675 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 08:47:05.561233 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 08:47:05.564831 13857 tablet.cc:2369] T 00000000000000000000000000000000 P c1e42147924247afac06c81128a3143c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:47:05.571558 13977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a40e6463cb8d459ea1b5717555390000: Bootstrap starting.
I20260928 08:47:05.575563 13977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a40e6463cb8d459ea1b5717555390000: Neither blocks nor log segments found. Creating new log.
I20260928 08:47:05.576138 14026 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3ea965677eaa450487b3fa6edb175f3b [term 1 FOLLOWER]: Refusing update from remote peer c1e42147924247afac06c81128a3143c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:47:05.576864 13977 log.cc:824] T 00000000000000000000000000000000 P a40e6463cb8d459ea1b5717555390000: Log is configured to *not* fsync() on all Append() calls
I20260928 08:47:05.578064 14091 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1e42147924247afac06c81128a3143c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ea965677eaa450487b3fa6edb175f3b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:47:05.604020 14064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ea965677eaa450487b3fa6edb175f3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1e42147924247afac06c81128a3143c. Latest consensus state: current_term: 1 leader_uuid: "c1e42147924247afac06c81128a3143c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e42147924247afac06c81128a3143c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34491 } } peers { permanent_uuid: "a40e6463cb8d459ea1b5717555390000" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45211 } } peers { permanent_uuid: "3ea965677eaa450487b3fa6edb175f3b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40857 } } }
I20260928 08:47:05.604367 14064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ea965677eaa450487b3fa6edb175f3b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:47:05.607617 14094 mvcc.cc:205] Tried to move back new op lower bound from 7334237083936239616 to 7334237083783684096. Current Snapshot: MvccSnapshot[applied={T|T < 7334237083936239616}]
I20260928 08:47:05.613677 14096 mvcc.cc:205] Tried to move back new op lower bound from 7334237083936239616 to 7334237083783684096. Current Snapshot: MvccSnapshot[applied={T|T < 7334237083936239616}]
I20260928 08:47:05.617830 14078 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1e42147924247afac06c81128a3143c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1e42147924247afac06c81128a3143c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e42147924247afac06c81128a3143c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34491 } } peers { permanent_uuid: "a40e6463cb8d459ea1b5717555390000" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45211 } } peers { permanent_uuid: "3ea965677eaa450487b3fa6edb175f3b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40857 } } }
I20260928 08:47:05.618197 14078 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1e42147924247afac06c81128a3143c [sys.catalog]: This master's current role is: LEADER
I20260928 08:47:05.619167 14064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ea965677eaa450487b3fa6edb175f3b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1e42147924247afac06c81128a3143c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e42147924247afac06c81128a3143c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34491 } } peers { permanent_uuid: "a40e6463cb8d459ea1b5717555390000" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45211 } } peers { permanent_uuid: "3ea965677eaa450487b3fa6edb175f3b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40857 } } }
I20260928 08:47:05.619431 14064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ea965677eaa450487b3fa6edb175f3b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:47:05.626251 14093 catalog_manager.cc:1347] Generated new cluster ID: b8816740747c4c6287a150087631071b
I20260928 08:47:05.626353 14093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:47:05.628829 13977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a40e6463cb8d459ea1b5717555390000: No bootstrap required, opened a new log
I20260928 08:47:05.651278 13977 raft_consensus.cc:348] T 00000000000000000000000000000000 P a40e6463cb8d459ea1b5717555390000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e42147924247afac06c81128a3143c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34491 } } peers { permanent_uuid: "a40e6463cb8d459ea1b5717555390000" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45211 } } peers { permanent_uuid: "3ea965677eaa450487b3fa6edb175f3b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40857 } }
I20260928 08:47:05.652347 13977 raft_consensus.cc:374] T 00000000000000000000000000000000 P a40e6463cb8d459ea1b5717555390000 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:47:05.652155 14093 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:47:05.652598 13977 raft_consensus.cc:729] T 00000000000000000000000000000000 P a40e6463cb8d459ea1b5717555390000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a40e6463cb8d459ea1b5717555390000, State: Initialized, Role: FOLLOWER
I20260928 08:47:05.654083 13977 consensus_queue.cc:260] T 00000000000000000000000000000000 P a40e6463cb8d459ea1b5717555390000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e42147924247afac06c81128a3143c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34491 } } peers { permanent_uuid: "a40e6463cb8d459ea1b5717555390000" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45211 } } peers { permanent_uuid: "3ea965677eaa450487b3fa6edb175f3b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40857 } }
I20260928 08:47:05.656363 14093 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:47:05.667572 14101 sys_catalog.cc:455] T 00000000000000000000000000000000 P a40e6463cb8d459ea1b5717555390000 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e42147924247afac06c81128a3143c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34491 } } peers { permanent_uuid: "a40e6463cb8d459ea1b5717555390000" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45211 } } peers { permanent_uuid: "3ea965677eaa450487b3fa6edb175f3b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40857 } } }
I20260928 08:47:05.668035 14101 sys_catalog.cc:458] T 00000000000000000000000000000000 P a40e6463cb8d459ea1b5717555390000 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:47:05.671140 13952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c1e42147924247afac06c81128a3143c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a40e6463cb8d459ea1b5717555390000" is_pre_election: true
I20260928 08:47:05.671612 13951 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c1e42147924247afac06c81128a3143c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a40e6463cb8d459ea1b5717555390000"
I20260928 08:47:05.672427 13951 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a40e6463cb8d459ea1b5717555390000 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:47:05.675848 13951 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a40e6463cb8d459ea1b5717555390000 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1e42147924247afac06c81128a3143c in term 1.
I20260928 08:47:05.678133 13950 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a40e6463cb8d459ea1b5717555390000 [term 1 FOLLOWER]: Refusing update from remote peer c1e42147924247afac06c81128a3143c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:47:05.679112 14078 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1e42147924247afac06c81128a3143c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a40e6463cb8d459ea1b5717555390000" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:47:05.679365 13952 raft_consensus.cc:2362] T 00000000000000000000000000000000 P a40e6463cb8d459ea1b5717555390000 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate c1e42147924247afac06c81128a3143c in term 1. Re-sending same reply.
I20260928 08:47:05.685458 13977 sys_catalog.cc:564] T 00000000000000000000000000000000 P a40e6463cb8d459ea1b5717555390000 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:47:05.685949 14093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1e42147924247afac06c81128a3143c: Generated new TSK 0
I20260928 08:47:05.686995 14093 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:47:05.732944 14101 sys_catalog.cc:455] T 00000000000000000000000000000000 P a40e6463cb8d459ea1b5717555390000 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1e42147924247afac06c81128a3143c. Latest consensus state: current_term: 1 leader_uuid: "c1e42147924247afac06c81128a3143c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e42147924247afac06c81128a3143c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34491 } } peers { permanent_uuid: "a40e6463cb8d459ea1b5717555390000" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45211 } } peers { permanent_uuid: "3ea965677eaa450487b3fa6edb175f3b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40857 } } }
I20260928 08:47:05.733351 14101 sys_catalog.cc:458] T 00000000000000000000000000000000 P a40e6463cb8d459ea1b5717555390000 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:47:05.732944 14108 sys_catalog.cc:455] T 00000000000000000000000000000000 P a40e6463cb8d459ea1b5717555390000 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1e42147924247afac06c81128a3143c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e42147924247afac06c81128a3143c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34491 } } peers { permanent_uuid: "a40e6463cb8d459ea1b5717555390000" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45211 } } peers { permanent_uuid: "3ea965677eaa450487b3fa6edb175f3b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40857 } } }
I20260928 08:47:05.733639 14108 sys_catalog.cc:458] T 00000000000000000000000000000000 P a40e6463cb8d459ea1b5717555390000 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:47:05.734983 14100 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1e42147924247afac06c81128a3143c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1e42147924247afac06c81128a3143c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c1e42147924247afac06c81128a3143c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34491 } } peers { permanent_uuid: "a40e6463cb8d459ea1b5717555390000" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 45211 } } peers { permanent_uuid: "3ea965677eaa450487b3fa6edb175f3b" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 40857 } } }
I20260928 08:47:05.735423 14100 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1e42147924247afac06c81128a3143c [sys.catalog]: This master's current role is: LEADER
I20260928 08:47:05.783232 14121 catalog_manager.cc:1259] Loaded cluster ID: b8816740747c4c6287a150087631071b
I20260928 08:47:05.783425 14121 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a40e6463cb8d459ea1b5717555390000: loading cluster ID for follower catalog manager: success
I20260928 08:47:05.787218 14121 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a40e6463cb8d459ea1b5717555390000: acquiring CA information for follower catalog manager: success
I20260928 08:47:05.789775 14121 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a40e6463cb8d459ea1b5717555390000: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 08:47:06.429136 14075 catalog_manager.cc:1259] Loaded cluster ID: b8816740747c4c6287a150087631071b
I20260928 08:47:06.429243 14075 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3ea965677eaa450487b3fa6edb175f3b: loading cluster ID for follower catalog manager: success
I20260928 08:47:06.431726 14075 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3ea965677eaa450487b3fa6edb175f3b: acquiring CA information for follower catalog manager: success
I20260928 08:47:06.433583 14075 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3ea965677eaa450487b3fa6edb175f3b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 08:47:06.823755 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 13836
I20260928 08:47:06.825136 13898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:47:06.936707 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 13836
W20260928 08:47:06.957362 13929 connection.cc:535] server connection from 127.0.0.1:33956 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 08:47:06.957891 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 13910
I20260928 08:47:06.958945 13972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:47:07.066332 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 13910
I20260928 08:47:07.086633 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 13984
I20260928 08:47:07.087891 14046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:47:07.195011 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 13984
2026-09-28T08:47:07Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4645 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:47:07 dist-test-slave-n6xs krb5kdc[14128](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:47:07 dist-test-slave-n6xs krb5kdc[14128](info): set up 2 sockets
Sep 28 08:47:07 dist-test-slave-n6xs krb5kdc[14128](info): commencing operation
krb5kdc: starting...
W20260928 08:47:08.304240 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:47:08 dist-test-slave-n6xs krb5kdc[14128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:47:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:47:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:47:08.533778 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38509
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:38509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:47:08.859701 14141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:47:08.860002 14141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:47:08.860091 14141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:47:08.867189 14141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:47:08.867292 14141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:47:08.867337 14141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:47:08.867385 14141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:47:08.867427 14141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:47:08.877040 14141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:38509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:38509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:47:08.878221 14141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:47:08.879449 14141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:47:08.891822 14146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:08.891871 14149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:08.891875 14147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:47:08.892751 14141 server_base.cc:1034] running on GCE node
I20260928 08:47:08.893504 14141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:47:08.894918 14141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:47:08.896153 14141 hybrid_clock.cc:648] HybridClock initialized: now 1790585228896120 us; error 61 us; skew 500 ppm
I20260928 08:47:08.896538 14141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:47:08 dist-test-slave-n6xs krb5kdc[14128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:47:08.899631 14141 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:47:08.901170 14141 webserver.cc:460] Webserver started at http://127.0.97.62:40049/ using document root <none> and password file <none>
I20260928 08:47:08.901816 14141 fs_manager.cc:359] Metadata directory not provided
I20260928 08:47:08.901912 14141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:47:08.902166 14141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:47:08.905150 14141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "b34fdf1bdb51413f94bbcb2ae33bc996"
format_stamp: "Formatted at 2026-09-28 08:47:08 on dist-test-slave-n6xs"
server_key: "d88809829276f5f8c32d9bc01b0f23ac"
server_key_iv: "df6a6c281dfd3f333884b338a734f715"
server_key_version: "encryptionkey@0"
I20260928 08:47:08.905829 14141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "b34fdf1bdb51413f94bbcb2ae33bc996"
format_stamp: "Formatted at 2026-09-28 08:47:08 on dist-test-slave-n6xs"
server_key: "d88809829276f5f8c32d9bc01b0f23ac"
server_key_iv: "df6a6c281dfd3f333884b338a734f715"
server_key_version: "encryptionkey@0"
I20260928 08:47:08.910869 14141 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260928 08:47:08.914158 14156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:08.915464 14141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:47:08.915598 14141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "b34fdf1bdb51413f94bbcb2ae33bc996"
format_stamp: "Formatted at 2026-09-28 08:47:08 on dist-test-slave-n6xs"
server_key: "d88809829276f5f8c32d9bc01b0f23ac"
server_key_iv: "df6a6c281dfd3f333884b338a734f715"
server_key_version: "encryptionkey@0"
I20260928 08:47:08.915761 14141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:47:08.927176 14141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:47:08.931523 14141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:47:08.931761 14141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:47:08.950519 14141 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:38509
I20260928 08:47:08.950551 14207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:38509 every 8 connection(s)
I20260928 08:47:08.952077 14141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:47:08.955848 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 14141
I20260928 08:47:08.956063 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:47:08.956185 14208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:47:08.956943 388 external_mini_cluster.cc:1428] Setting key f2a223a8b85cdfd2e907b1ea31250986
I20260928 08:47:08.968614 14208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b34fdf1bdb51413f94bbcb2ae33bc996: Bootstrap starting.
Sep 28 08:47:08 dist-test-slave-n6xs krb5kdc[14128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:47:08.971580 14208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b34fdf1bdb51413f94bbcb2ae33bc996: Neither blocks nor log segments found. Creating new log.
I20260928 08:47:08.972687 14208 log.cc:824] T 00000000000000000000000000000000 P b34fdf1bdb51413f94bbcb2ae33bc996: Log is configured to *not* fsync() on all Append() calls
I20260928 08:47:08.975404 14208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b34fdf1bdb51413f94bbcb2ae33bc996: No bootstrap required, opened a new log
I20260928 08:47:08.978794 14211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:08.958843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37162 (local address 127.0.97.62:38509)
0928 08:47:08.959655 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:08.959670 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:08.959717 (+ 47us) server_negotiation.cc:407] Connection header received
0928 08:47:08.961473 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:08.961494 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:08.961893 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:08.962289 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:08.963774 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:08.965113 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:08.966450 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:08.967162 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:08.967368 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:08.971507 (+ 4139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:08.971515 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:08.971529 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:08.971563 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:08.974786 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:08.975503 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:08.975511 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:08.975525 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:08.975680 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:08.976108 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:08.976113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:08.976117 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:08.976617 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:08.976948 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:08.977429 (+ 481us) server_negotiation.cc:299] Negotiation successful
0928 08:47:08.977826 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":379,"threads_started":1}
I20260928 08:47:08.982667 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:47:08.984115 14208 raft_consensus.cc:348] T 00000000000000000000000000000000 P b34fdf1bdb51413f94bbcb2ae33bc996 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34fdf1bdb51413f94bbcb2ae33bc996" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38509 } }
I20260928 08:47:08.984467 14208 raft_consensus.cc:374] T 00000000000000000000000000000000 P b34fdf1bdb51413f94bbcb2ae33bc996 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:47:08.984566 14208 raft_consensus.cc:729] T 00000000000000000000000000000000 P b34fdf1bdb51413f94bbcb2ae33bc996 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b34fdf1bdb51413f94bbcb2ae33bc996, State: Initialized, Role: FOLLOWER
I20260928 08:47:08.985198 14208 consensus_queue.cc:260] T 00000000000000000000000000000000 P b34fdf1bdb51413f94bbcb2ae33bc996 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34fdf1bdb51413f94bbcb2ae33bc996" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38509 } }
I20260928 08:47:08.985474 14208 raft_consensus.cc:388] T 00000000000000000000000000000000 P b34fdf1bdb51413f94bbcb2ae33bc996 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:47:08.985610 14208 raft_consensus.cc:482] T 00000000000000000000000000000000 P b34fdf1bdb51413f94bbcb2ae33bc996 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:47:08.985791 14208 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b34fdf1bdb51413f94bbcb2ae33bc996 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:47:08.987877 14208 raft_consensus.cc:504] T 00000000000000000000000000000000 P b34fdf1bdb51413f94bbcb2ae33bc996 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34fdf1bdb51413f94bbcb2ae33bc996" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38509 } }
I20260928 08:47:08.988315 14208 leader_election.cc:302] T 00000000000000000000000000000000 P b34fdf1bdb51413f94bbcb2ae33bc996 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b34fdf1bdb51413f94bbcb2ae33bc996; no voters:
I20260928 08:47:08.988848 14208 leader_election.cc:288] T 00000000000000000000000000000000 P b34fdf1bdb51413f94bbcb2ae33bc996 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:47:08.988984 14213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b34fdf1bdb51413f94bbcb2ae33bc996 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:47:08.989724 14213 raft_consensus.cc:686] T 00000000000000000000000000000000 P b34fdf1bdb51413f94bbcb2ae33bc996 [term 1 LEADER]: Becoming Leader. State: Replica: b34fdf1bdb51413f94bbcb2ae33bc996, State: Running, Role: LEADER
I20260928 08:47:08.990279 14213 consensus_queue.cc:237] T 00000000000000000000000000000000 P b34fdf1bdb51413f94bbcb2ae33bc996 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34fdf1bdb51413f94bbcb2ae33bc996" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38509 } }
I20260928 08:47:08.990995 14208 sys_catalog.cc:564] T 00000000000000000000000000000000 P b34fdf1bdb51413f94bbcb2ae33bc996 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:47:08.993372 14215 sys_catalog.cc:455] T 00000000000000000000000000000000 P b34fdf1bdb51413f94bbcb2ae33bc996 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b34fdf1bdb51413f94bbcb2ae33bc996. Latest consensus state: current_term: 1 leader_uuid: "b34fdf1bdb51413f94bbcb2ae33bc996" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34fdf1bdb51413f94bbcb2ae33bc996" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38509 } } }
I20260928 08:47:08.993758 14215 sys_catalog.cc:458] T 00000000000000000000000000000000 P b34fdf1bdb51413f94bbcb2ae33bc996 [sys.catalog]: This master's current role is: LEADER
I20260928 08:47:08.994495 14214 sys_catalog.cc:455] T 00000000000000000000000000000000 P b34fdf1bdb51413f94bbcb2ae33bc996 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b34fdf1bdb51413f94bbcb2ae33bc996" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b34fdf1bdb51413f94bbcb2ae33bc996" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 38509 } } }
I20260928 08:47:08.994837 14214 sys_catalog.cc:458] T 00000000000000000000000000000000 P b34fdf1bdb51413f94bbcb2ae33bc996 [sys.catalog]: This master's current role is: LEADER
I20260928 08:47:08.996066 14220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:47:09.001415 14220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:47:09.010154 14220 catalog_manager.cc:1347] Generated new cluster ID: 383ccd6b66ca4eb4b555e425d88e4a19
I20260928 08:47:09.010269 14220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:47:09.042953 14220 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:47:09.044445 14220 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:47:09.052699 14220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b34fdf1bdb51413f94bbcb2ae33bc996: Generated new TSK 0
I20260928 08:47:09.053574 14220 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:47:09.094890 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 14141
I20260928 08:47:09.096182 14203 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:47:09.192227 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 14141
2026-09-28T08:47:09Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1993 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:47:09 dist-test-slave-n6xs krb5kdc[14236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:47:09 dist-test-slave-n6xs krb5kdc[14236](info): set up 2 sockets
Sep 28 08:47:09 dist-test-slave-n6xs krb5kdc[14236](info): commencing operation
krb5kdc: starting...
W20260928 08:47:10.404351 388 mini_kdc.cc:121] Time spent starting KDC: real 1.174s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:47:10 dist-test-slave-n6xs krb5kdc[14236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:47:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:47:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:47:10.644308 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41033
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41033
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:47:10.987341 14249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:47:10.987656 14249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:47:10.987758 14249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:47:10.995639 14249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:47:10.995766 14249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:47:10.995890 14249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:47:10.995996 14249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:47:10.996048 14249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:47:11.006035 14249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:47:11.007393 14249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:47:11.008720 14249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:47:11.020691 14255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:11.020761 14257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:11.020735 14254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:47:11.021981 14249 server_base.cc:1034] running on GCE node
I20260928 08:47:11.022801 14249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:47:11.024235 14249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:47:11.025432 14249 hybrid_clock.cc:648] HybridClock initialized: now 1790585231025375 us; error 53 us; skew 500 ppm
I20260928 08:47:11.025772 14249 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:47:11 dist-test-slave-n6xs krb5kdc[14236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:47:11.032629 14249 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:47:11.034096 14249 webserver.cc:460] Webserver started at http://127.0.97.62:42203/ using document root <none> and password file <none>
I20260928 08:47:11.034739 14249 fs_manager.cc:359] Metadata directory not provided
I20260928 08:47:11.034837 14249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:47:11.035079 14249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:47:11.038079 14249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "aa24014e8c13464f8070bdccea5f1dd5"
format_stamp: "Formatted at 2026-09-28 08:47:11 on dist-test-slave-n6xs"
server_key: "1d6526fab231a8c99f1fca0056a36a8b"
server_key_iv: "5461d6bc2d753e9db4be248c2d951949"
server_key_version: "encryptionkey@0"
I20260928 08:47:11.038786 14249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "aa24014e8c13464f8070bdccea5f1dd5"
format_stamp: "Formatted at 2026-09-28 08:47:11 on dist-test-slave-n6xs"
server_key: "1d6526fab231a8c99f1fca0056a36a8b"
server_key_iv: "5461d6bc2d753e9db4be248c2d951949"
server_key_version: "encryptionkey@0"
I20260928 08:47:11.043784 14249 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 08:47:11.047204 14264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:11.048625 14249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 08:47:11.048818 14249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "aa24014e8c13464f8070bdccea5f1dd5"
format_stamp: "Formatted at 2026-09-28 08:47:11 on dist-test-slave-n6xs"
server_key: "1d6526fab231a8c99f1fca0056a36a8b"
server_key_iv: "5461d6bc2d753e9db4be248c2d951949"
server_key_version: "encryptionkey@0"
I20260928 08:47:11.048988 14249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:47:11.065948 14249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:47:11.069835 14249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:47:11.070119 14249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:47:11.088790 14249 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41033
I20260928 08:47:11.088824 14315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41033 every 8 connection(s)
I20260928 08:47:11.090193 14249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:47:11.093784 14316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:47:11.097848 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 14249
I20260928 08:47:11.098059 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:47:11.098737 388 external_mini_cluster.cc:1428] Setting key 374f0cd0981b82e3b535e02a7c8940a1
I20260928 08:47:11.105710 14316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5: Bootstrap starting.
I20260928 08:47:11.109316 14316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5: Neither blocks nor log segments found. Creating new log.
I20260928 08:47:11.110587 14316 log.cc:824] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5: Log is configured to *not* fsync() on all Append() calls
I20260928 08:47:11.113467 14316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5: No bootstrap required, opened a new log
Sep 28 08:47:11 dist-test-slave-n6xs krb5kdc[14236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:47:11.123773 14319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:11.100744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57354 (local address 127.0.97.62:41033)
0928 08:47:11.101546 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:11.101562 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:11.101616 (+ 54us) server_negotiation.cc:407] Connection header received
0928 08:47:11.104124 (+ 2508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:11.104151 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:11.104629 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:11.105056 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:11.106346 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:11.108377 (+ 2031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:11.109722 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:11.110799 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:11.110966 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:11.115070 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:11.115081 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:11.115105 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:11.115152 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:11.119216 (+ 4064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:11.119885 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:11.119895 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:11.119905 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:11.120116 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:11.120495 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:11.120500 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:11.120503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:11.121229 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:11.121488 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:11.121897 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 08:47:11.122355 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":312,"threads_started":1}
I20260928 08:47:11.124341 14316 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa24014e8c13464f8070bdccea5f1dd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41033 } }
I20260928 08:47:11.124714 14316 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:47:11.124817 14316 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa24014e8c13464f8070bdccea5f1dd5, State: Initialized, Role: FOLLOWER
I20260928 08:47:11.125526 14316 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa24014e8c13464f8070bdccea5f1dd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41033 } }
I20260928 08:47:11.125818 14316 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:47:11.125955 14316 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:47:11.126140 14316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:47:11.128587 14316 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa24014e8c13464f8070bdccea5f1dd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41033 } }
I20260928 08:47:11.129021 14316 leader_election.cc:302] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa24014e8c13464f8070bdccea5f1dd5; no voters:
I20260928 08:47:11.129581 14316 leader_election.cc:288] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:47:11.129770 14321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:47:11.130623 14321 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [term 1 LEADER]: Becoming Leader. State: Replica: aa24014e8c13464f8070bdccea5f1dd5, State: Running, Role: LEADER
I20260928 08:47:11.131156 14321 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa24014e8c13464f8070bdccea5f1dd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41033 } }
I20260928 08:47:11.132710 14316 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:47:11.134110 14322 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa24014e8c13464f8070bdccea5f1dd5. Latest consensus state: current_term: 1 leader_uuid: "aa24014e8c13464f8070bdccea5f1dd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa24014e8c13464f8070bdccea5f1dd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41033 } } }
I20260928 08:47:11.134205 14323 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa24014e8c13464f8070bdccea5f1dd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa24014e8c13464f8070bdccea5f1dd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41033 } } }
I20260928 08:47:11.134639 14323 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [sys.catalog]: This master's current role is: LEADER
I20260928 08:47:11.134676 14322 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [sys.catalog]: This master's current role is: LEADER
I20260928 08:47:11.136793 14326 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:47:11.142273 14326 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:47:11.150945 14326 catalog_manager.cc:1347] Generated new cluster ID: 3fd605c65cc848b98489e0243a4b9e58
I20260928 08:47:11.151052 14326 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:47:11.175027 14326 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:47:11.176786 14326 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:47:11.193097 14326 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5: Generated new TSK 0
I20260928 08:47:11.194227 14326 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:47:11.255587 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41033
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:47:11.587368 14342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:47:11.587687 14342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:47:11.587893 14342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:47:11.595060 14342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:47:11.595197 14342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:47:11.595372 14342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:47:11.606189 14342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41033
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:47:11.607471 14342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:47:11.608945 14342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:47:11.621887 14350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:11.621860 14348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:11.622081 14347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:47:11.624111 14342 server_base.cc:1034] running on GCE node
I20260928 08:47:11.624740 14342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:47:11.625674 14342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:47:11.626909 14342 hybrid_clock.cc:648] HybridClock initialized: now 1790585231626874 us; error 45 us; skew 500 ppm
I20260928 08:47:11.627261 14342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:47:11 dist-test-slave-n6xs krb5kdc[14236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:47:11.630492 14342 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:47:11.631996 14342 webserver.cc:460] Webserver started at http://127.0.97.1:34731/ using document root <none> and password file <none>
I20260928 08:47:11.632684 14342 fs_manager.cc:359] Metadata directory not provided
I20260928 08:47:11.632815 14342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:47:11.633131 14342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:47:11.636328 14342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/data/instance:
uuid: "8604a55c5af4410194362b5517fcf5da"
format_stamp: "Formatted at 2026-09-28 08:47:11 on dist-test-slave-n6xs"
server_key: "8ae245d09be6f937dc3b85b1db1e7425"
server_key_iv: "baa2e661e39fb911be6fcdb1261eab03"
server_key_version: "encryptionkey@0"
I20260928 08:47:11.637043 14342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/wal/instance:
uuid: "8604a55c5af4410194362b5517fcf5da"
format_stamp: "Formatted at 2026-09-28 08:47:11 on dist-test-slave-n6xs"
server_key: "8ae245d09be6f937dc3b85b1db1e7425"
server_key_iv: "baa2e661e39fb911be6fcdb1261eab03"
server_key_version: "encryptionkey@0"
I20260928 08:47:11.642190 14342 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:47:11.645879 14357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:11.647145 14342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:47:11.647303 14342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/wal
uuid: "8604a55c5af4410194362b5517fcf5da"
format_stamp: "Formatted at 2026-09-28 08:47:11 on dist-test-slave-n6xs"
server_key: "8ae245d09be6f937dc3b85b1db1e7425"
server_key_iv: "baa2e661e39fb911be6fcdb1261eab03"
server_key_version: "encryptionkey@0"
I20260928 08:47:11.647504 14342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:47:11.664950 14342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:47:11.668635 14342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:47:11.668910 14342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:47:11.669823 14342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:47:11.671111 14342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:47:11.671185 14342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:11.671343 14342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:47:11.671404 14342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:11.704798 14342 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35491
I20260928 08:47:11.704891 14469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35491 every 8 connection(s)
I20260928 08:47:11.706082 14342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:47:11.711341 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 14342
I20260928 08:47:11.711549 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:47:11.712368 388 external_mini_cluster.cc:1428] Setting key a0c86ffab1ccd31df611af9bf1345e0f
Sep 28 08:47:11 dist-test-slave-n6xs krb5kdc[14236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:47:11.729130 14472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:11.708858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:46603 (local address 127.0.97.62:41033)
0928 08:47:11.709348 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:11.709354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:11.710331 (+ 977us) server_negotiation.cc:407] Connection header received
0928 08:47:11.712810 (+ 2479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:11.712815 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:11.712886 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:11.712973 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:11.714781 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:11.715485 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:11.718077 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:11.718524 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:11.718651 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:11.722983 (+ 4332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:11.722996 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:11.722999 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:11.723034 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:11.725734 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:11.726575 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:11.726580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:11.726582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:11.726708 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:11.727176 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:11.727189 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:11.727191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:11.727482 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:11.727651 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:11.728462 (+ 811us) server_negotiation.cc:299] Negotiation successful
0928 08:47:11.728708 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":234,"threads_started":1}
I20260928 08:47:11.730155 14473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:11.709359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.1:46603)
0928 08:47:11.710167 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:11.710211 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:11.712471 (+ 2260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:11.713219 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:11.713228 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:11.713825 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:47:11.714581 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:11.714626 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:11.715646 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:11.715654 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:11.717892 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:11.717909 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:11.718758 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:11.718768 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:11.719112 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:11.719935 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:47:11.719967 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:47:11.722748 (+ 2781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:47:11.725906 (+ 3158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:11.725916 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:11.725932 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:11.726375 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:11.726835 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:11.726841 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:11.726845 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:11.727050 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:11.727623 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:47:11.727631 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:47:11.728050 (+ 419us) client_negotiation.cc:769] Sending connection context
0928 08:47:11.728333 (+ 283us) client_negotiation.cc:240] Negotiation successful
0928 08:47:11.728895 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":324,"threads_started":1}
I20260928 08:47:11.731895 14470 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41033
I20260928 08:47:11.732283 14470 heartbeater.cc:464] Registering TS with master...
I20260928 08:47:11.733120 14470 heartbeater.cc:511] Master 127.0.97.62:41033 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 08:47:11.735684 14281 ts_manager.cc:194] Registered new tserver with Master: 8604a55c5af4410194362b5517fcf5da (127.0.97.1:35491)
I20260928 08:47:11.737213 14281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:46603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 08:47:11.767331 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41033
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:47:12.095278 14476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:47:12.095597 14476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:47:12.095728 14476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:47:12.107189 14476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:47:12.107328 14476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:47:12.107501 14476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:47:12.117487 14476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41033
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:47:12.118844 14476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:47:12.120225 14476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:47:12.133769 14484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:12.133903 14481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:12.133913 14482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:47:12.137297 14476 server_base.cc:1034] running on GCE node
I20260928 08:47:12.138020 14476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:47:12.139142 14476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:47:12.140339 14476 hybrid_clock.cc:648] HybridClock initialized: now 1790585232140297 us; error 37 us; skew 500 ppm
I20260928 08:47:12.140656 14476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:47:12 dist-test-slave-n6xs krb5kdc[14236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:47:12.143740 14476 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:47:12.145434 14476 webserver.cc:460] Webserver started at http://127.0.97.2:40595/ using document root <none> and password file <none>
I20260928 08:47:12.146103 14476 fs_manager.cc:359] Metadata directory not provided
I20260928 08:47:12.146198 14476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:47:12.146445 14476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:47:12.149391 14476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/data/instance:
uuid: "53c708b11d6142ba939ac6ace881b828"
format_stamp: "Formatted at 2026-09-28 08:47:12 on dist-test-slave-n6xs"
server_key: "f4b72c65f9705c2d14dd9b775a1a442f"
server_key_iv: "0aeadb982c5c08c7540275da145ca85c"
server_key_version: "encryptionkey@0"
I20260928 08:47:12.150133 14476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/wal/instance:
uuid: "53c708b11d6142ba939ac6ace881b828"
format_stamp: "Formatted at 2026-09-28 08:47:12 on dist-test-slave-n6xs"
server_key: "f4b72c65f9705c2d14dd9b775a1a442f"
server_key_iv: "0aeadb982c5c08c7540275da145ca85c"
server_key_version: "encryptionkey@0"
I20260928 08:47:12.155406 14476 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260928 08:47:12.158818 14491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:12.160293 14476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:47:12.160459 14476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/wal
uuid: "53c708b11d6142ba939ac6ace881b828"
format_stamp: "Formatted at 2026-09-28 08:47:12 on dist-test-slave-n6xs"
server_key: "f4b72c65f9705c2d14dd9b775a1a442f"
server_key_iv: "0aeadb982c5c08c7540275da145ca85c"
server_key_version: "encryptionkey@0"
I20260928 08:47:12.160715 14476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:47:12.197875 14476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:47:12.202242 14476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:47:12.202535 14476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:47:12.203536 14476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:47:12.204918 14476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:47:12.205044 14476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:12.205178 14476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:47:12.205246 14476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:12.240067 14476 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41805
I20260928 08:47:12.240149 14603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41805 every 8 connection(s)
I20260928 08:47:12.241343 14476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:47:12.250262 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 14476
I20260928 08:47:12.250438 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:47:12.250896 388 external_mini_cluster.cc:1428] Setting key de9d064fd35a76073ef7b15d70306e05
Sep 28 08:47:12 dist-test-slave-n6xs krb5kdc[14236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:47:12.262207 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:12.243855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:38225 (local address 127.0.97.62:41033)
0928 08:47:12.244298 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:12.244303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:12.245282 (+ 979us) server_negotiation.cc:407] Connection header received
0928 08:47:12.247147 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:12.247155 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:12.247251 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:12.247369 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:12.248942 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:12.249671 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:12.251585 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:12.252237 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:12.252382 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:12.256433 (+ 4051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:12.256445 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:12.256448 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:12.256480 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:12.259032 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:12.259751 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:12.259756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:12.259760 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:12.259948 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:12.260385 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:12.260390 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:12.260393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:12.260728 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:12.260872 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:12.261723 (+ 851us) server_negotiation.cc:299] Negotiation successful
0928 08:47:12.261897 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":216,"threads_started":1}
I20260928 08:47:12.263693 14607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:12.244465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.2:38225)
0928 08:47:12.245130 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:12.245171 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:12.246882 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:12.247630 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:12.247637 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:12.248209 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:47:12.248766 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:12.248787 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:12.249883 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:12.249887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:12.251416 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:12.251428 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:12.252462 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:12.252469 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:12.252751 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:12.253479 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:47:12.253513 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:47:12.256220 (+ 2707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:47:12.259184 (+ 2964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:12.259192 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:12.259207 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:12.259629 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:12.260087 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:12.260092 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:12.260096 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:12.260269 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:12.260900 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:47:12.260911 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:47:12.261328 (+ 417us) client_negotiation.cc:769] Sending connection context
0928 08:47:12.261627 (+ 299us) client_negotiation.cc:240] Negotiation successful
0928 08:47:12.262200 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":262,"threads_started":1}
I20260928 08:47:12.265239 14604 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41033
I20260928 08:47:12.265573 14604 heartbeater.cc:464] Registering TS with master...
I20260928 08:47:12.266269 14604 heartbeater.cc:511] Master 127.0.97.62:41033 requested a full tablet report, sending...
I20260928 08:47:12.267731 14281 ts_manager.cc:194] Registered new tserver with Master: 53c708b11d6142ba939ac6ace881b828 (127.0.97.2:41805)
I20260928 08:47:12.268558 14281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:38225
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:47:12.303188 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41033
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:47:12.627601 14610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:47:12.627969 14610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:47:12.628072 14610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:47:12.635466 14610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:47:12.635569 14610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:47:12.635680 14610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:47:12.645509 14610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:41033
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:47:12.646714 14610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:47:12.647972 14610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:47:12.662005 14616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:12.662091 14615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:12.662102 14618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:47:12.662729 14610 server_base.cc:1034] running on GCE node
I20260928 08:47:12.663340 14610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:47:12.664403 14610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:47:12.665601 14610 hybrid_clock.cc:648] HybridClock initialized: now 1790585232665537 us; error 52 us; skew 500 ppm
I20260928 08:47:12.665912 14610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:47:12 dist-test-slave-n6xs krb5kdc[14236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:47:12.669006 14610 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:47:12.670329 14610 webserver.cc:460] Webserver started at http://127.0.97.3:33867/ using document root <none> and password file <none>
I20260928 08:47:12.670938 14610 fs_manager.cc:359] Metadata directory not provided
I20260928 08:47:12.671051 14610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:47:12.671276 14610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:47:12.674101 14610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/data/instance:
uuid: "90ba2aa9a3be471b92e79e27031c753d"
format_stamp: "Formatted at 2026-09-28 08:47:12 on dist-test-slave-n6xs"
server_key: "29149447c5ef103043d690b75402b5a2"
server_key_iv: "bb933ea3b91bf56175b56018d33da89b"
server_key_version: "encryptionkey@0"
I20260928 08:47:12.674796 14610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/wal/instance:
uuid: "90ba2aa9a3be471b92e79e27031c753d"
format_stamp: "Formatted at 2026-09-28 08:47:12 on dist-test-slave-n6xs"
server_key: "29149447c5ef103043d690b75402b5a2"
server_key_iv: "bb933ea3b91bf56175b56018d33da89b"
server_key_version: "encryptionkey@0"
I20260928 08:47:12.679668 14610 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 08:47:12.683043 14625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:12.684422 14610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:47:12.684594 14610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/wal
uuid: "90ba2aa9a3be471b92e79e27031c753d"
format_stamp: "Formatted at 2026-09-28 08:47:12 on dist-test-slave-n6xs"
server_key: "29149447c5ef103043d690b75402b5a2"
server_key_iv: "bb933ea3b91bf56175b56018d33da89b"
server_key_version: "encryptionkey@0"
I20260928 08:47:12.684798 14610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:47:12.714553 14610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:47:12.718586 14610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:47:12.718878 14610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:47:12.719910 14610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:47:12.721153 14610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:47:12.721256 14610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:12.721407 14610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:47:12.721479 14610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:12.740357 14470 heartbeater.cc:503] Master 127.0.97.62:41033 was elected leader, sending a full tablet report...
I20260928 08:47:12.754817 14610 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43385
I20260928 08:47:12.754856 14737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43385 every 8 connection(s)
I20260928 08:47:12.756233 14610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:47:12.764786 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 14610
I20260928 08:47:12.764985 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:47:12.765506 388 external_mini_cluster.cc:1428] Setting key 033ebe6defc53a1a69fcba9d7e289f88
Sep 28 08:47:12 dist-test-slave-n6xs krb5kdc[14236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:47:12.776795 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:12.758746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:57703 (local address 127.0.97.62:41033)
0928 08:47:12.758981 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:12.758987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:12.760182 (+ 1195us) server_negotiation.cc:407] Connection header received
0928 08:47:12.762063 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:12.762068 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:12.762154 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:12.762244 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:12.763672 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:12.764488 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:12.766421 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:12.767066 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:12.767168 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:12.771181 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:12.771186 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:12.771190 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:12.771233 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:12.773784 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:12.774542 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:12.774546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:12.774548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:12.774669 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:12.775146 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:12.775149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:12.775151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:12.775416 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:12.775533 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:12.776375 (+ 842us) server_negotiation.cc:299] Negotiation successful
0928 08:47:12.776556 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 08:47:12.777849 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:12.759292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.3:57703)
0928 08:47:12.760028 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:12.760066 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:12.761790 (+ 1724us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:12.762439 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:12.762447 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:12.762955 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:47:12.763520 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:12.763538 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:12.764747 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:12.764754 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:12.766259 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:12.766273 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:12.767327 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:12.767347 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:12.767643 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:12.768521 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:47:12.768546 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:47:12.770975 (+ 2429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:47:12.773938 (+ 2963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:12.773945 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:12.773960 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:12.774424 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:12.774815 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:12.774819 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:12.774822 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:12.775032 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:12.775535 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:47:12.775541 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:47:12.775978 (+ 437us) client_negotiation.cc:769] Sending connection context
0928 08:47:12.776259 (+ 281us) client_negotiation.cc:240] Negotiation successful
0928 08:47:12.776675 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":287,"threads_started":1}
I20260928 08:47:12.779349 14738 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41033
I20260928 08:47:12.779695 14738 heartbeater.cc:464] Registering TS with master...
I20260928 08:47:12.780622 14738 heartbeater.cc:511] Master 127.0.97.62:41033 requested a full tablet report, sending...
I20260928 08:47:12.782191 14281 ts_manager.cc:194] Registered new tserver with Master: 90ba2aa9a3be471b92e79e27031c753d (127.0.97.3:43385)
I20260928 08:47:12.782961 14281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:57703
I20260928 08:47:12.793839 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:47:12.811484 14606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:12.798694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57356 (local address 127.0.97.62:41033)
0928 08:47:12.798936 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:12.798943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:12.799269 (+ 326us) server_negotiation.cc:407] Connection header received
0928 08:47:12.799476 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:12.799481 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:12.799544 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:12.799654 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:12.801086 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:12.801817 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:12.803264 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:12.803686 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:12.803843 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:12.806128 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:12.806133 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:12.806136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:12.806167 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:12.808801 (+ 2634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:12.809500 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:12.809506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:12.809509 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:12.809632 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:12.810024 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:12.810029 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:12.810033 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:12.810341 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:12.810620 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:12.810904 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 08:47:12.811155 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 08:47:12.815976 14281 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57356:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:47:12.823074 14281 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:47:12.850889 14751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:12.835788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57310 (local address 127.0.97.1:35491)
0928 08:47:12.837673 (+ 1885us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:12.837681 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:12.837746 (+ 65us) server_negotiation.cc:407] Connection header received
0928 08:47:12.837878 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:12.837889 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:12.838282 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:12.838492 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:47:12.840746 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:12.843717 (+ 2971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:12.848372 (+ 4655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:12.849615 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:12.849802 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:12.850062 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:12.850324 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 08:47:12.850485 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1702,"thread_start_us":4774,"threads_started":1}
I20260928 08:47:12.850895 14750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:12.835517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35491 (local address 127.0.0.1:57310)
0928 08:47:12.836415 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:12.836475 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:12.836661 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:12.839942 (+ 3281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:12.839953 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:12.840002 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:47:12.840527 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:12.840543 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:12.843888 (+ 3345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:12.843896 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:12.848142 (+ 4246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:12.848163 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:12.849837 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:12.849843 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:12.850046 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:12.850109 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 08:47:12.850276 (+ 167us) client_negotiation.cc:240] Negotiation successful
0928 08:47:12.850486 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":383,"threads_started":1}
I20260928 08:47:12.855798 14405 tablet_service.cc:1467] Processing CreateTablet for tablet 3aa067dcf4994962963f2ac8f698d36c (DEFAULT_TABLE table=test-table [id=4d3acbda58cb45c39289758c0b8752f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:47:12.857237 14405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aa067dcf4994962963f2ac8f698d36c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:47:12.859807 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:12.838015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57360 (local address 127.0.97.2:41805)
0928 08:47:12.849199 (+ 11184us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:12.849216 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:12.851010 (+ 1794us) server_negotiation.cc:407] Connection header received
0928 08:47:12.851164 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:12.851178 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:12.851558 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:12.851860 (+ 302us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:47:12.853147 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:12.854657 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:12.857159 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:12.858636 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:12.858764 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:12.858963 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:12.859325 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 08:47:12.859489 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10932,"thread_start_us":10733,"threads_started":1}
I20260928 08:47:12.859897 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:12.837375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41805 (local address 127.0.0.1:57360)
0928 08:47:12.850595 (+ 13220us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:12.850628 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:12.850810 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:12.852229 (+ 1419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:12.852238 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:12.852283 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:47:12.852857 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:12.852872 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:12.854818 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:12.854827 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:12.856978 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:12.856994 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:12.858874 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:12.858885 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:12.859097 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:12.859144 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 08:47:12.859246 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 08:47:12.859406 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13033,"thread_start_us":8963,"threads_started":1}
I20260928 08:47:12.861829 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:12.836932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43385 (local address 127.0.0.1:50458)
0928 08:47:12.846755 (+ 9823us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:12.846785 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:12.846952 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:12.854064 (+ 7112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:12.854073 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:12.854103 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:47:12.854504 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:12.854516 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:12.856462 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:12.856468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:12.858773 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:12.858797 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:12.860857 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:12.860866 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:12.861078 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:12.861128 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 08:47:12.861270 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 08:47:12.861421 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9621,"thread_start_us":14446,"threads_started":1}
I20260928 08:47:12.862106 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:12.837375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50458 (local address 127.0.97.3:43385)
0928 08:47:12.853339 (+ 15964us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:12.853347 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:12.853379 (+ 32us) server_negotiation.cc:407] Connection header received
0928 08:47:12.853471 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:12.853478 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:12.853696 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:12.854049 (+ 353us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:47:12.854702 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:12.856292 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:12.859017 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:12.860617 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:12.860796 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:12.861080 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:12.861312 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 08:47:12.861490 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15791,"thread_start_us":15622,"threads_started":1}
I20260928 08:47:12.863533 14539 tablet_service.cc:1467] Processing CreateTablet for tablet 3aa067dcf4994962963f2ac8f698d36c (DEFAULT_TABLE table=test-table [id=4d3acbda58cb45c39289758c0b8752f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:47:12.865119 14539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aa067dcf4994962963f2ac8f698d36c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:47:12.867430 14673 tablet_service.cc:1467] Processing CreateTablet for tablet 3aa067dcf4994962963f2ac8f698d36c (DEFAULT_TABLE table=test-table [id=4d3acbda58cb45c39289758c0b8752f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:47:12.868818 14673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aa067dcf4994962963f2ac8f698d36c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:47:12.873296 14756 tablet_bootstrap.cc:492] T 3aa067dcf4994962963f2ac8f698d36c P 8604a55c5af4410194362b5517fcf5da: Bootstrap starting.
I20260928 08:47:12.878425 14757 tablet_bootstrap.cc:492] T 3aa067dcf4994962963f2ac8f698d36c P 53c708b11d6142ba939ac6ace881b828: Bootstrap starting.
I20260928 08:47:12.878787 14756 tablet_bootstrap.cc:654] T 3aa067dcf4994962963f2ac8f698d36c P 8604a55c5af4410194362b5517fcf5da: Neither blocks nor log segments found. Creating new log.
I20260928 08:47:12.880286 14756 log.cc:824] T 3aa067dcf4994962963f2ac8f698d36c P 8604a55c5af4410194362b5517fcf5da: Log is configured to *not* fsync() on all Append() calls
I20260928 08:47:12.881510 14757 tablet_bootstrap.cc:654] T 3aa067dcf4994962963f2ac8f698d36c P 53c708b11d6142ba939ac6ace881b828: Neither blocks nor log segments found. Creating new log.
I20260928 08:47:12.882417 14757 log.cc:824] T 3aa067dcf4994962963f2ac8f698d36c P 53c708b11d6142ba939ac6ace881b828: Log is configured to *not* fsync() on all Append() calls
I20260928 08:47:12.883760 14756 tablet_bootstrap.cc:492] T 3aa067dcf4994962963f2ac8f698d36c P 8604a55c5af4410194362b5517fcf5da: No bootstrap required, opened a new log
I20260928 08:47:12.884095 14756 ts_tablet_manager.cc:1397] T 3aa067dcf4994962963f2ac8f698d36c P 8604a55c5af4410194362b5517fcf5da: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260928 08:47:12.885018 14757 tablet_bootstrap.cc:492] T 3aa067dcf4994962963f2ac8f698d36c P 53c708b11d6142ba939ac6ace881b828: No bootstrap required, opened a new log
I20260928 08:47:12.885284 14757 ts_tablet_manager.cc:1397] T 3aa067dcf4994962963f2ac8f698d36c P 53c708b11d6142ba939ac6ace881b828: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 08:47:12.885530 14760 tablet_bootstrap.cc:492] T 3aa067dcf4994962963f2ac8f698d36c P 90ba2aa9a3be471b92e79e27031c753d: Bootstrap starting.
I20260928 08:47:12.888885 14760 tablet_bootstrap.cc:654] T 3aa067dcf4994962963f2ac8f698d36c P 90ba2aa9a3be471b92e79e27031c753d: Neither blocks nor log segments found. Creating new log.
I20260928 08:47:12.890182 14760 log.cc:824] T 3aa067dcf4994962963f2ac8f698d36c P 90ba2aa9a3be471b92e79e27031c753d: Log is configured to *not* fsync() on all Append() calls
I20260928 08:47:12.893311 14760 tablet_bootstrap.cc:492] T 3aa067dcf4994962963f2ac8f698d36c P 90ba2aa9a3be471b92e79e27031c753d: No bootstrap required, opened a new log
I20260928 08:47:12.893560 14760 ts_tablet_manager.cc:1397] T 3aa067dcf4994962963f2ac8f698d36c P 90ba2aa9a3be471b92e79e27031c753d: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260928 08:47:12.894403 14756 raft_consensus.cc:348] T 3aa067dcf4994962963f2ac8f698d36c P 8604a55c5af4410194362b5517fcf5da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8604a55c5af4410194362b5517fcf5da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35491 } } peers { permanent_uuid: "90ba2aa9a3be471b92e79e27031c753d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43385 } } peers { permanent_uuid: "53c708b11d6142ba939ac6ace881b828" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41805 } }
I20260928 08:47:12.894853 14756 raft_consensus.cc:374] T 3aa067dcf4994962963f2ac8f698d36c P 8604a55c5af4410194362b5517fcf5da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:47:12.895002 14756 raft_consensus.cc:729] T 3aa067dcf4994962963f2ac8f698d36c P 8604a55c5af4410194362b5517fcf5da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8604a55c5af4410194362b5517fcf5da, State: Initialized, Role: FOLLOWER
I20260928 08:47:12.895328 14757 raft_consensus.cc:348] T 3aa067dcf4994962963f2ac8f698d36c P 53c708b11d6142ba939ac6ace881b828 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8604a55c5af4410194362b5517fcf5da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35491 } } peers { permanent_uuid: "90ba2aa9a3be471b92e79e27031c753d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43385 } } peers { permanent_uuid: "53c708b11d6142ba939ac6ace881b828" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41805 } }
I20260928 08:47:12.895850 14757 raft_consensus.cc:374] T 3aa067dcf4994962963f2ac8f698d36c P 53c708b11d6142ba939ac6ace881b828 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:47:12.896044 14757 raft_consensus.cc:729] T 3aa067dcf4994962963f2ac8f698d36c P 53c708b11d6142ba939ac6ace881b828 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c708b11d6142ba939ac6ace881b828, State: Initialized, Role: FOLLOWER
I20260928 08:47:12.895913 14756 consensus_queue.cc:260] T 3aa067dcf4994962963f2ac8f698d36c P 8604a55c5af4410194362b5517fcf5da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8604a55c5af4410194362b5517fcf5da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35491 } } peers { permanent_uuid: "90ba2aa9a3be471b92e79e27031c753d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43385 } } peers { permanent_uuid: "53c708b11d6142ba939ac6ace881b828" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41805 } }
I20260928 08:47:12.896682 14757 consensus_queue.cc:260] T 3aa067dcf4994962963f2ac8f698d36c P 53c708b11d6142ba939ac6ace881b828 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8604a55c5af4410194362b5517fcf5da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35491 } } peers { permanent_uuid: "90ba2aa9a3be471b92e79e27031c753d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43385 } } peers { permanent_uuid: "53c708b11d6142ba939ac6ace881b828" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41805 } }
I20260928 08:47:12.898139 14604 heartbeater.cc:503] Master 127.0.97.62:41033 was elected leader, sending a full tablet report...
I20260928 08:47:12.898517 14757 ts_tablet_manager.cc:1428] T 3aa067dcf4994962963f2ac8f698d36c P 53c708b11d6142ba939ac6ace881b828: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260928 08:47:12.900231 14756 ts_tablet_manager.cc:1428] T 3aa067dcf4994962963f2ac8f698d36c P 8604a55c5af4410194362b5517fcf5da: Time spent starting tablet: real 0.016s user 0.014s sys 0.001s
I20260928 08:47:12.904812 14760 raft_consensus.cc:348] T 3aa067dcf4994962963f2ac8f698d36c P 90ba2aa9a3be471b92e79e27031c753d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8604a55c5af4410194362b5517fcf5da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35491 } } peers { permanent_uuid: "90ba2aa9a3be471b92e79e27031c753d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43385 } } peers { permanent_uuid: "53c708b11d6142ba939ac6ace881b828" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41805 } }
I20260928 08:47:12.905169 14760 raft_consensus.cc:374] T 3aa067dcf4994962963f2ac8f698d36c P 90ba2aa9a3be471b92e79e27031c753d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:47:12.905297 14760 raft_consensus.cc:729] T 3aa067dcf4994962963f2ac8f698d36c P 90ba2aa9a3be471b92e79e27031c753d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90ba2aa9a3be471b92e79e27031c753d, State: Initialized, Role: FOLLOWER
I20260928 08:47:12.905853 14760 consensus_queue.cc:260] T 3aa067dcf4994962963f2ac8f698d36c P 90ba2aa9a3be471b92e79e27031c753d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8604a55c5af4410194362b5517fcf5da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35491 } } peers { permanent_uuid: "90ba2aa9a3be471b92e79e27031c753d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43385 } } peers { permanent_uuid: "53c708b11d6142ba939ac6ace881b828" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41805 } }
I20260928 08:47:12.906811 14738 heartbeater.cc:503] Master 127.0.97.62:41033 was elected leader, sending a full tablet report...
I20260928 08:47:12.907393 14760 ts_tablet_manager.cc:1428] T 3aa067dcf4994962963f2ac8f698d36c P 90ba2aa9a3be471b92e79e27031c753d: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260928 08:47:12.922210 14764 raft_consensus.cc:482] T 3aa067dcf4994962963f2ac8f698d36c P 90ba2aa9a3be471b92e79e27031c753d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:47:12.922452 14764 raft_consensus.cc:504] T 3aa067dcf4994962963f2ac8f698d36c P 90ba2aa9a3be471b92e79e27031c753d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8604a55c5af4410194362b5517fcf5da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35491 } } peers { permanent_uuid: "90ba2aa9a3be471b92e79e27031c753d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43385 } } peers { permanent_uuid: "53c708b11d6142ba939ac6ace881b828" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41805 } }
I20260928 08:47:12.924079 14764 leader_election.cc:288] T 3aa067dcf4994962963f2ac8f698d36c P 90ba2aa9a3be471b92e79e27031c753d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8604a55c5af4410194362b5517fcf5da (127.0.97.1:35491), 53c708b11d6142ba939ac6ace881b828 (127.0.97.2:41805)
I20260928 08:47:12.930764 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:12.924449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35491 (local address 127.0.97.3:51079)
0928 08:47:12.924842 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:12.924863 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:12.925024 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:12.925419 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:12.925423 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:12.925445 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:47:12.925738 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:12.925750 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:12.926978 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:12.926983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:12.928572 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:12.928584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:12.929938 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:12.929945 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:12.930136 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:12.930171 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 08:47:12.930275 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 08:47:12.930413 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":22,"spinlock_wait_cycles":23808}
I20260928 08:47:12.930822 14751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:12.924469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:51079 (local address 127.0.97.1:35491)
0928 08:47:12.924701 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:12.924708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:12.924953 (+ 245us) server_negotiation.cc:407] Connection header received
0928 08:47:12.925200 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:12.925206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:12.925288 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:12.925392 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:47:12.925898 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:12.926843 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:12.928728 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:12.929778 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:12.929883 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:12.929957 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:12.930360 (+ 403us) server_negotiation.cc:299] Negotiation successful
0928 08:47:12.930517 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 08:47:12.931586 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:12.924533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:49927 (local address 127.0.97.2:41805)
0928 08:47:12.924716 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:12.924723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:12.925517 (+ 794us) server_negotiation.cc:407] Connection header received
0928 08:47:12.925693 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:12.925700 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:12.925770 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:12.925858 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:47:12.926465 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:12.927329 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:12.929342 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:12.930396 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:12.930519 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:12.930614 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:12.931114 (+ 500us) server_negotiation.cc:299] Negotiation successful
0928 08:47:12.931273 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 08:47:12.931931 14425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aa067dcf4994962963f2ac8f698d36c" candidate_uuid: "90ba2aa9a3be471b92e79e27031c753d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8604a55c5af4410194362b5517fcf5da" is_pre_election: true
I20260928 08:47:12.932327 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:12.924570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41805 (local address 127.0.97.3:49927)
0928 08:47:12.925408 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:12.925426 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:12.925556 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:12.925888 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:12.925892 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:12.925915 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:47:12.926318 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:12.926331 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:12.927475 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:12.927483 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:12.929187 (+ 1704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:12.929203 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:12.930631 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:12.930639 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:12.930866 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:12.930909 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 08:47:12.931796 (+ 887us) client_negotiation.cc:240] Negotiation successful
0928 08:47:12.931970 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"mutex_wait_us":1,"thread_start_us":528,"threads_started":1}
I20260928 08:47:12.932514 14425 raft_consensus.cc:2454] T 3aa067dcf4994962963f2ac8f698d36c P 8604a55c5af4410194362b5517fcf5da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90ba2aa9a3be471b92e79e27031c753d in term 0.
I20260928 08:47:12.933293 14626 leader_election.cc:302] T 3aa067dcf4994962963f2ac8f698d36c P 90ba2aa9a3be471b92e79e27031c753d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8604a55c5af4410194362b5517fcf5da, 90ba2aa9a3be471b92e79e27031c753d; no voters:
I20260928 08:47:12.933658 14764 raft_consensus.cc:2781] T 3aa067dcf4994962963f2ac8f698d36c P 90ba2aa9a3be471b92e79e27031c753d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:47:12.933542 14559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aa067dcf4994962963f2ac8f698d36c" candidate_uuid: "90ba2aa9a3be471b92e79e27031c753d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53c708b11d6142ba939ac6ace881b828" is_pre_election: true
I20260928 08:47:12.933847 14764 raft_consensus.cc:482] T 3aa067dcf4994962963f2ac8f698d36c P 90ba2aa9a3be471b92e79e27031c753d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:47:12.933959 14764 raft_consensus.cc:3037] T 3aa067dcf4994962963f2ac8f698d36c P 90ba2aa9a3be471b92e79e27031c753d [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:47:12.934230 14559 raft_consensus.cc:2454] T 3aa067dcf4994962963f2ac8f698d36c P 53c708b11d6142ba939ac6ace881b828 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90ba2aa9a3be471b92e79e27031c753d in term 0.
I20260928 08:47:12.936606 14764 raft_consensus.cc:504] T 3aa067dcf4994962963f2ac8f698d36c P 90ba2aa9a3be471b92e79e27031c753d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8604a55c5af4410194362b5517fcf5da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35491 } } peers { permanent_uuid: "90ba2aa9a3be471b92e79e27031c753d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43385 } } peers { permanent_uuid: "53c708b11d6142ba939ac6ace881b828" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41805 } }
I20260928 08:47:12.937294 14764 leader_election.cc:288] T 3aa067dcf4994962963f2ac8f698d36c P 90ba2aa9a3be471b92e79e27031c753d [CANDIDATE]: Term 1 election: Requested vote from peers 8604a55c5af4410194362b5517fcf5da (127.0.97.1:35491), 53c708b11d6142ba939ac6ace881b828 (127.0.97.2:41805)
I20260928 08:47:12.937670 14425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aa067dcf4994962963f2ac8f698d36c" candidate_uuid: "90ba2aa9a3be471b92e79e27031c753d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8604a55c5af4410194362b5517fcf5da"
I20260928 08:47:12.937816 14559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aa067dcf4994962963f2ac8f698d36c" candidate_uuid: "90ba2aa9a3be471b92e79e27031c753d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53c708b11d6142ba939ac6ace881b828"
I20260928 08:47:12.937889 14425 raft_consensus.cc:3037] T 3aa067dcf4994962963f2ac8f698d36c P 8604a55c5af4410194362b5517fcf5da [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:47:12.938110 14559 raft_consensus.cc:3037] T 3aa067dcf4994962963f2ac8f698d36c P 53c708b11d6142ba939ac6ace881b828 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:47:12.940172 14425 raft_consensus.cc:2454] T 3aa067dcf4994962963f2ac8f698d36c P 8604a55c5af4410194362b5517fcf5da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90ba2aa9a3be471b92e79e27031c753d in term 1.
I20260928 08:47:12.940479 14559 raft_consensus.cc:2454] T 3aa067dcf4994962963f2ac8f698d36c P 53c708b11d6142ba939ac6ace881b828 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90ba2aa9a3be471b92e79e27031c753d in term 1.
I20260928 08:47:12.940521 14626 leader_election.cc:302] T 3aa067dcf4994962963f2ac8f698d36c P 90ba2aa9a3be471b92e79e27031c753d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8604a55c5af4410194362b5517fcf5da, 90ba2aa9a3be471b92e79e27031c753d; no voters:
I20260928 08:47:12.940837 14764 raft_consensus.cc:2781] T 3aa067dcf4994962963f2ac8f698d36c P 90ba2aa9a3be471b92e79e27031c753d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:47:12.941314 14764 raft_consensus.cc:686] T 3aa067dcf4994962963f2ac8f698d36c P 90ba2aa9a3be471b92e79e27031c753d [term 1 LEADER]: Becoming Leader. State: Replica: 90ba2aa9a3be471b92e79e27031c753d, State: Running, Role: LEADER
I20260928 08:47:12.941838 14764 consensus_queue.cc:237] T 3aa067dcf4994962963f2ac8f698d36c P 90ba2aa9a3be471b92e79e27031c753d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8604a55c5af4410194362b5517fcf5da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35491 } } peers { permanent_uuid: "90ba2aa9a3be471b92e79e27031c753d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43385 } } peers { permanent_uuid: "53c708b11d6142ba939ac6ace881b828" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41805 } }
I20260928 08:47:12.946306 14280 catalog_manager.cc:5495] T 3aa067dcf4994962963f2ac8f698d36c P 90ba2aa9a3be471b92e79e27031c753d reported cstate change: term changed from 0 to 1, leader changed from <none> to 90ba2aa9a3be471b92e79e27031c753d (127.0.97.3). New cstate: current_term: 1 leader_uuid: "90ba2aa9a3be471b92e79e27031c753d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8604a55c5af4410194362b5517fcf5da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90ba2aa9a3be471b92e79e27031c753d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53c708b11d6142ba939ac6ace881b828" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41805 } health_report { overall_health: UNKNOWN } } }
W20260928 08:47:12.958425 14471 tablet.cc:2369] T 3aa067dcf4994962963f2ac8f698d36c P 8604a55c5af4410194362b5517fcf5da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:47:12.974661 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:12.969573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50466 (local address 127.0.97.3:43385)
0928 08:47:12.969804 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:12.969808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:12.969826 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:47:12.969911 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:12.969914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:12.969991 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:12.970135 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:47:12.970445 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:12.971172 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:12.972564 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:12.972963 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:12.973062 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:12.973533 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:47:12.974041 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:47:12.974163 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:12.974269 (+ 106us) server_negotiation.cc:299] Negotiation successful
0928 08:47:12.974341 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 08:47:12.982712 14559 raft_consensus.cc:1261] T 3aa067dcf4994962963f2ac8f698d36c P 53c708b11d6142ba939ac6ace881b828 [term 1 FOLLOWER]: Refusing update from remote peer 90ba2aa9a3be471b92e79e27031c753d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:47:12.982776 14425 raft_consensus.cc:1261] T 3aa067dcf4994962963f2ac8f698d36c P 8604a55c5af4410194362b5517fcf5da [term 1 FOLLOWER]: Refusing update from remote peer 90ba2aa9a3be471b92e79e27031c753d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:47:12.983712 14766 consensus_queue.cc:1035] T 3aa067dcf4994962963f2ac8f698d36c P 90ba2aa9a3be471b92e79e27031c753d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8604a55c5af4410194362b5517fcf5da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:47:12.984108 14764 consensus_queue.cc:1035] T 3aa067dcf4994962963f2ac8f698d36c P 90ba2aa9a3be471b92e79e27031c753d [LEADER]: Connected to new peer: Peer: permanent_uuid: "53c708b11d6142ba939ac6ace881b828" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 08:47:12.993311 14605 tablet.cc:2369] T 3aa067dcf4994962963f2ac8f698d36c P 53c708b11d6142ba939ac6ace881b828: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:47:12.996724 14770 mvcc.cc:205] Tried to move back new op lower bound from 7334237114287308800 to 7334237114133958656. Current Snapshot: MvccSnapshot[applied={T|T < 7334237114287308800}]
I20260928 08:47:13.011299 14280 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:47:13.011598 14280 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:47:13.015419 14280 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5: Sending DeleteTablet for 3 replicas of tablet 3aa067dcf4994962963f2ac8f698d36c
I20260928 08:47:13.016597 14405 tablet_service.cc:1514] Processing DeleteTablet for tablet 3aa067dcf4994962963f2ac8f698d36c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:47:13 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57310
I20260928 08:47:13.016659 14673 tablet_service.cc:1514] Processing DeleteTablet for tablet 3aa067dcf4994962963f2ac8f698d36c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:47:13 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50458
I20260928 08:47:13.017091 14539 tablet_service.cc:1514] Processing DeleteTablet for tablet 3aa067dcf4994962963f2ac8f698d36c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:47:13 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57360
I20260928 08:47:13.018077 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 14342
I20260928 08:47:13.029130 14780 tablet_replica.cc:330] T 3aa067dcf4994962963f2ac8f698d36c P 8604a55c5af4410194362b5517fcf5da: stopping tablet replica
I20260928 08:47:13.030531 14780 raft_consensus.cc:2229] T 3aa067dcf4994962963f2ac8f698d36c P 8604a55c5af4410194362b5517fcf5da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:47:13.032017 14780 raft_consensus.cc:2258] T 3aa067dcf4994962963f2ac8f698d36c P 8604a55c5af4410194362b5517fcf5da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:47:13.032344 14782 tablet_replica.cc:330] T 3aa067dcf4994962963f2ac8f698d36c P 53c708b11d6142ba939ac6ace881b828: stopping tablet replica
I20260928 08:47:13.032811 14782 raft_consensus.cc:2229] T 3aa067dcf4994962963f2ac8f698d36c P 53c708b11d6142ba939ac6ace881b828 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:47:13.033366 14782 raft_consensus.cc:2258] T 3aa067dcf4994962963f2ac8f698d36c P 53c708b11d6142ba939ac6ace881b828 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:47:13.034852 14781 tablet_replica.cc:330] T 3aa067dcf4994962963f2ac8f698d36c P 90ba2aa9a3be471b92e79e27031c753d: stopping tablet replica
I20260928 08:47:13.035395 14781 raft_consensus.cc:2229] T 3aa067dcf4994962963f2ac8f698d36c P 90ba2aa9a3be471b92e79e27031c753d [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:47:13.036049 14780 ts_tablet_manager.cc:1905] T 3aa067dcf4994962963f2ac8f698d36c P 8604a55c5af4410194362b5517fcf5da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:47:13.036245 14781 raft_consensus.cc:2258] T 3aa067dcf4994962963f2ac8f698d36c P 90ba2aa9a3be471b92e79e27031c753d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:47:13.038340 14781 ts_tablet_manager.cc:1905] T 3aa067dcf4994962963f2ac8f698d36c P 90ba2aa9a3be471b92e79e27031c753d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:47:13.038880 14782 ts_tablet_manager.cc:1905] T 3aa067dcf4994962963f2ac8f698d36c P 53c708b11d6142ba939ac6ace881b828: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:47:13.043466 14780 ts_tablet_manager.cc:1918] T 3aa067dcf4994962963f2ac8f698d36c P 8604a55c5af4410194362b5517fcf5da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:47:13.043679 14780 log.cc:1196] T 3aa067dcf4994962963f2ac8f698d36c P 8604a55c5af4410194362b5517fcf5da: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/wal/wals/3aa067dcf4994962963f2ac8f698d36c
I20260928 08:47:13.044270 14780 ts_tablet_manager.cc:1939] T 3aa067dcf4994962963f2ac8f698d36c P 8604a55c5af4410194362b5517fcf5da: Deleting consensus metadata
I20260928 08:47:13.045845 14781 ts_tablet_manager.cc:1918] T 3aa067dcf4994962963f2ac8f698d36c P 90ba2aa9a3be471b92e79e27031c753d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:47:13.045787 14782 ts_tablet_manager.cc:1918] T 3aa067dcf4994962963f2ac8f698d36c P 53c708b11d6142ba939ac6ace881b828: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:47:13.046164 14782 log.cc:1196] T 3aa067dcf4994962963f2ac8f698d36c P 53c708b11d6142ba939ac6ace881b828: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/wal/wals/3aa067dcf4994962963f2ac8f698d36c
I20260928 08:47:13.046137 14265 catalog_manager.cc:4841] TS 8604a55c5af4410194362b5517fcf5da (127.0.97.1:35491): tablet 3aa067dcf4994962963f2ac8f698d36c (table test-table [id=4d3acbda58cb45c39289758c0b8752f3]) successfully deleted
I20260928 08:47:13.046443 14781 log.cc:1196] T 3aa067dcf4994962963f2ac8f698d36c P 90ba2aa9a3be471b92e79e27031c753d: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/wal/wals/3aa067dcf4994962963f2ac8f698d36c
I20260928 08:47:13.046715 14782 ts_tablet_manager.cc:1939] T 3aa067dcf4994962963f2ac8f698d36c P 53c708b11d6142ba939ac6ace881b828: Deleting consensus metadata
I20260928 08:47:13.047070 14781 ts_tablet_manager.cc:1939] T 3aa067dcf4994962963f2ac8f698d36c P 90ba2aa9a3be471b92e79e27031c753d: Deleting consensus metadata
I20260928 08:47:13.048507 14267 catalog_manager.cc:4841] TS 53c708b11d6142ba939ac6ace881b828 (127.0.97.2:41805): tablet 3aa067dcf4994962963f2ac8f698d36c (table test-table [id=4d3acbda58cb45c39289758c0b8752f3]) successfully deleted
I20260928 08:47:13.049165 14265 catalog_manager.cc:4841] TS 90ba2aa9a3be471b92e79e27031c753d (127.0.97.3:43385): tablet 3aa067dcf4994962963f2ac8f698d36c (table test-table [id=4d3acbda58cb45c39289758c0b8752f3]) successfully deleted
Sep 28 08:47:13 dist-test-slave-n6xs krb5kdc[14236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:47:13.058632 14751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:13.019618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57314 (local address 127.0.97.1:35491)
0928 08:47:13.019891 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:13.019900 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:13.029397 (+ 9497us) server_negotiation.cc:407] Connection header received
0928 08:47:13.029680 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:13.029710 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:13.029815 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:13.029894 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:13.032653 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:13.034127 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:13.036269 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:13.036957 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:13.037073 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:13.051559 (+ 14486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:13.051576 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:13.051595 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:13.051639 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:13.055507 (+ 3868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:13.056408 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:13.056418 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:13.056422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:13.056582 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:13.056985 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:13.056990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:13.056994 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:13.057427 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:13.057756 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:13.058009 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 08:47:13.058244 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 08:47:13.059451 14465 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:47:13.163152 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 14342
I20260928 08:47:13.181483 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 14476
Sep 28 08:47:13 dist-test-slave-n6xs krb5kdc[14236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 08:47:13.199194 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:13.182631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57362 (local address 127.0.97.2:41805)
0928 08:47:13.182829 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:13.182833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:13.182850 (+ 17us) server_negotiation.cc:407] Connection header received
0928 08:47:13.182901 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:13.182905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:13.182964 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:13.183057 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:13.184775 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:13.185513 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:13.187397 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:13.187794 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:13.188128 (+ 334us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:13.193394 (+ 5266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:13.193408 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:13.193424 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:13.193458 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:13.196511 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:13.197331 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:13.197338 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:13.197340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:13.197453 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:13.197830 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:13.197833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:13.197835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:13.198194 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:13.198400 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:13.198760 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 08:47:13.198913 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 08:47:13.199923 14599 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:47:13.289129 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 14476
I20260928 08:47:13.313104 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 14610
Sep 28 08:47:13 dist-test-slave-n6xs krb5kdc[14236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 08:47:13.330792 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:13.313898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50472 (local address 127.0.97.3:43385)
0928 08:47:13.314108 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:13.314114 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:13.314136 (+ 22us) server_negotiation.cc:407] Connection header received
0928 08:47:13.314284 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:13.314295 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:13.314386 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:13.314525 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:13.315756 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:13.317103 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:13.318478 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:13.319159 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:13.319293 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:13.323573 (+ 4280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:13.323590 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:13.323612 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:13.323662 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:13.327751 (+ 4089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:13.328465 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:13.328475 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:13.328479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:13.328668 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:13.329020 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:13.329025 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:13.329028 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:13.329504 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:13.329857 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:13.330136 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 08:47:13.330364 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 08:47:13.331681 14733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:47:13.458705 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 14610
I20260928 08:47:13.478474 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 14249
I20260928 08:47:13.479617 14311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:47:13.593359 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 14249
I20260928 08:47:13.616040 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41033
--webserver_interface=127.0.97.62
--webserver_port=42203
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:47:13.944545 14788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:47:13.944840 14788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:47:13.944996 14788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:47:13.952376 14788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:47:13.952487 14788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:47:13.952564 14788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:47:13.952657 14788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:47:13.952754 14788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:47:13.963006 14788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=42203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:47:13.964488 14788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:47:13.965864 14788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:47:13.989122 14794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:13.989223 14797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:13.989218 14795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:47:13.990173 14788 server_base.cc:1034] running on GCE node
I20260928 08:47:13.991081 14788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:47:13.992833 14788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:47:13.994108 14788 hybrid_clock.cc:648] HybridClock initialized: now 1790585233994056 us; error 42 us; skew 500 ppm
I20260928 08:47:13.994521 14788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:47:13 dist-test-slave-n6xs krb5kdc[14236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:47:13.998471 14788 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:47:14.000384 14788 webserver.cc:460] Webserver started at http://127.0.97.62:42203/ using document root <none> and password file <none>
I20260928 08:47:14.001192 14788 fs_manager.cc:359] Metadata directory not provided
I20260928 08:47:14.001359 14788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:47:14.007386 14788 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 08:47:14.010576 14804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:14.012259 14788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:47:14.012462 14788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "aa24014e8c13464f8070bdccea5f1dd5"
format_stamp: "Formatted at 2026-09-28 08:47:11 on dist-test-slave-n6xs"
server_key: "1d6526fab231a8c99f1fca0056a36a8b"
server_key_iv: "5461d6bc2d753e9db4be248c2d951949"
server_key_version: "encryptionkey@0"
I20260928 08:47:14.013590 14788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:47:14.037822 14788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:47:14.042483 14788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:47:14.042775 14788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:47:14.062676 14788 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41033
I20260928 08:47:14.062712 14855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41033 every 8 connection(s)
I20260928 08:47:14.064316 14788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:47:14.069460 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 14788
I20260928 08:47:14.070264 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:35491
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:41033
--webserver_port=34731
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:47:14.071403 14856 sys_catalog.cc:263] Verifying existing consensus state
I20260928 08:47:14.074673 14856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5: Bootstrap starting.
I20260928 08:47:14.094019 14856 log.cc:824] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5: Log is configured to *not* fsync() on all Append() calls
I20260928 08:47:14.106216 14856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:47:14.106772 14856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5: Bootstrap complete.
I20260928 08:47:14.115887 14856 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa24014e8c13464f8070bdccea5f1dd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41033 } }
I20260928 08:47:14.116712 14856 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa24014e8c13464f8070bdccea5f1dd5, State: Initialized, Role: FOLLOWER
I20260928 08:47:14.117388 14856 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa24014e8c13464f8070bdccea5f1dd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41033 } }
I20260928 08:47:14.117658 14856 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:47:14.117784 14856 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:47:14.117971 14856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [term 1 FOLLOWER]: Advancing to term 2
I20260928 08:47:14.121923 14856 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa24014e8c13464f8070bdccea5f1dd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41033 } }
I20260928 08:47:14.122453 14856 leader_election.cc:302] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa24014e8c13464f8070bdccea5f1dd5; no voters:
I20260928 08:47:14.123034 14856 leader_election.cc:288] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 08:47:14.123240 14860 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 08:47:14.124449 14860 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [term 2 LEADER]: Becoming Leader. State: Replica: aa24014e8c13464f8070bdccea5f1dd5, State: Running, Role: LEADER
I20260928 08:47:14.125089 14860 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa24014e8c13464f8070bdccea5f1dd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41033 } }
I20260928 08:47:14.125247 14856 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:47:14.127683 14862 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa24014e8c13464f8070bdccea5f1dd5. Latest consensus state: current_term: 2 leader_uuid: "aa24014e8c13464f8070bdccea5f1dd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa24014e8c13464f8070bdccea5f1dd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41033 } } }
I20260928 08:47:14.128182 14862 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [sys.catalog]: This master's current role is: LEADER
I20260928 08:47:14.128940 14861 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "aa24014e8c13464f8070bdccea5f1dd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa24014e8c13464f8070bdccea5f1dd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41033 } } }
I20260928 08:47:14.129285 14861 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [sys.catalog]: This master's current role is: LEADER
I20260928 08:47:14.131207 14865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:47:14.140445 14865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:47:14.142326 14865 catalog_manager.cc:1259] Loaded cluster ID: 3fd605c65cc848b98489e0243a4b9e58
I20260928 08:47:14.142524 14865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:47:14.149392 14865 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 08:47:14.175058 14865 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 08:47:14.188158 14865 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:47:14.190582 14865 catalog_manager.cc:5879] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5: Loaded TSK: 0
I20260928 08:47:14.192721 14865 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 08:47:14.429483 14858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:47:14.429845 14858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:47:14.430035 14858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:47:14.437912 14858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:47:14.438079 14858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:47:14.438271 14858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:47:14.448589 14858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:35491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=34731
--tserver_master_addrs=127.0.97.62:41033
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:47:14.450053 14858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:47:14.451490 14858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:47:14.465590 14884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:14.466214 14883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:14.467031 14886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:47:14.467629 14858 server_base.cc:1034] running on GCE node
I20260928 08:47:14.468374 14858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:47:14.469432 14858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:47:14.470672 14858 hybrid_clock.cc:648] HybridClock initialized: now 1790585234470618 us; error 47 us; skew 500 ppm
I20260928 08:47:14.471063 14858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:47:14 dist-test-slave-n6xs krb5kdc[14236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:47:14.474632 14858 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:47:14.476519 14858 webserver.cc:460] Webserver started at http://127.0.97.1:34731/ using document root <none> and password file <none>
I20260928 08:47:14.477276 14858 fs_manager.cc:359] Metadata directory not provided
I20260928 08:47:14.477428 14858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:47:14.483400 14858 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260928 08:47:14.486552 14893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:14.488261 14858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 08:47:14.488471 14858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/wal
uuid: "8604a55c5af4410194362b5517fcf5da"
format_stamp: "Formatted at 2026-09-28 08:47:11 on dist-test-slave-n6xs"
server_key: "8ae245d09be6f937dc3b85b1db1e7425"
server_key_iv: "baa2e661e39fb911be6fcdb1261eab03"
server_key_version: "encryptionkey@0"
I20260928 08:47:14.489550 14858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:47:14.527758 14858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:47:14.532545 14858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:47:14.532869 14858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:47:14.534070 14858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:47:14.535595 14858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:47:14.535691 14858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:14.535862 14858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:47:14.535946 14858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:14.571798 14858 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35491
I20260928 08:47:14.571931 15005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35491 every 8 connection(s)
I20260928 08:47:14.573418 14858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:47:14.573891 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 14858
I20260928 08:47:14.574668 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:41805
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:41033
--webserver_port=40595
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:47:14 dist-test-slave-n6xs krb5kdc[14236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:47:14.599987 15008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:14.577137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.1:47011)
0928 08:47:14.577986 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:14.578083 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:14.580036 (+ 1953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:14.581551 (+ 1515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:14.581562 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:14.582227 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:47:14.582791 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:14.582810 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:14.584743 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:14.584747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:14.586009 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:14.586023 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:14.587100 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:14.587107 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:14.587389 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:14.588215 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:47:14.588243 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:47:14.590852 (+ 2609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:47:14.594934 (+ 4082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:14.594946 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:14.594966 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:14.595515 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:14.596193 (+ 678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:14.596200 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:14.596205 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:14.596431 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:14.597675 (+ 1244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:47:14.597681 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:47:14.598031 (+ 350us) client_negotiation.cc:769] Sending connection context
0928 08:47:14.598298 (+ 267us) client_negotiation.cc:240] Negotiation successful
0928 08:47:14.598765 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":746,"threads_started":1}
I20260928 08:47:14.600232 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:14.577835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:47011 (local address 127.0.97.62:41033)
0928 08:47:14.578497 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:14.578512 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:14.578567 (+ 55us) server_negotiation.cc:407] Connection header received
0928 08:47:14.580614 (+ 2047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:14.580628 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:14.581000 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:14.581386 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:14.583045 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:14.584563 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:14.586170 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:14.586843 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:14.587085 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:14.591227 (+ 4142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:14.591237 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:14.591255 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:14.591299 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:14.594760 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:14.595727 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:14.595744 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:14.595751 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:14.596054 (+ 303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:14.596607 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:14.596616 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:14.596622 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:14.597464 (+ 842us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:14.597714 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:14.598560 (+ 846us) server_negotiation.cc:299] Negotiation successful
0928 08:47:14.599022 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":1019,"threads_started":1}
I20260928 08:47:14.603415 15006 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41033
I20260928 08:47:14.603794 15006 heartbeater.cc:464] Registering TS with master...
I20260928 08:47:14.604664 15006 heartbeater.cc:511] Master 127.0.97.62:41033 requested a full tablet report, sending...
I20260928 08:47:14.607051 14821 ts_manager.cc:194] Registered new tserver with Master: 8604a55c5af4410194362b5517fcf5da (127.0.97.1:35491)
I20260928 08:47:14.608711 14821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:47011
W20260928 08:47:14.919697 15010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:47:14.920142 15010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:47:14.920325 15010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:47:14.928311 15010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:47:14.928503 15010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:47:14.928697 15010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:47:14.939601 15010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:41805
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=40595
--tserver_master_addrs=127.0.97.62:41033
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:47:14.941092 15010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:47:14.942602 15010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:47:14.957324 15015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:14.957288 15018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:14.957376 15016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:47:14.958832 15010 server_base.cc:1034] running on GCE node
I20260928 08:47:14.959502 15010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:47:14.960592 15010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:47:14.961817 15010 hybrid_clock.cc:648] HybridClock initialized: now 1790585234961741 us; error 78 us; skew 500 ppm
I20260928 08:47:14.962127 15010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:47:14 dist-test-slave-n6xs krb5kdc[14236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:47:14.965744 15010 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:47:14.967428 15010 webserver.cc:460] Webserver started at http://127.0.97.2:40595/ using document root <none> and password file <none>
I20260928 08:47:14.968346 15010 fs_manager.cc:359] Metadata directory not provided
I20260928 08:47:14.968472 15010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:47:14.974472 15010 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 08:47:14.977540 15025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:14.979100 15010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:47:14.979292 15010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/wal
uuid: "53c708b11d6142ba939ac6ace881b828"
format_stamp: "Formatted at 2026-09-28 08:47:12 on dist-test-slave-n6xs"
server_key: "f4b72c65f9705c2d14dd9b775a1a442f"
server_key_iv: "0aeadb982c5c08c7540275da145ca85c"
server_key_version: "encryptionkey@0"
I20260928 08:47:14.980413 15010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:47:15.015925 15010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:47:15.020951 15010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:47:15.021277 15010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:47:15.022396 15010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:47:15.023905 15010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:47:15.024050 15010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:15.024181 15010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:47:15.024286 15010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:15.060585 15010 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41805
I20260928 08:47:15.060614 15137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41805 every 8 connection(s)
I20260928 08:47:15.062325 15010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:47:15.070935 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 15010
I20260928 08:47:15.071632 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:43385
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:41033
--webserver_port=33867
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:47:15 dist-test-slave-n6xs krb5kdc[14236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:47:15.091500 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:15.065209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34887 (local address 127.0.97.62:41033)
0928 08:47:15.065435 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:15.065442 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:15.066857 (+ 1415us) server_negotiation.cc:407] Connection header received
0928 08:47:15.069563 (+ 2706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:15.069568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:15.069686 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:15.069788 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:15.072119 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.073356 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.075885 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.076692 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.076851 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:15.083071 (+ 6220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:15.083081 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:15.083089 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:15.083140 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:15.087108 (+ 3968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:15.088318 (+ 1210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:15.088327 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:15.088334 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:15.088532 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:15.089091 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:15.089100 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:15.089106 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:15.089537 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:15.089729 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:15.090687 (+ 958us) server_negotiation.cc:299] Negotiation successful
0928 08:47:15.091002 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 08:47:15.092198 15140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:15.065739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.2:34887)
0928 08:47:15.066642 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:15.066706 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:15.069196 (+ 2490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:15.070114 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:15.070127 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:15.070927 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:47:15.071777 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:15.071847 (+ 70us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.073589 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.073595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:15.075652 (+ 2057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:15.075668 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.076982 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.076991 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:15.077350 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:15.078219 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:47:15.078252 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:47:15.082780 (+ 4528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:47:15.087336 (+ 4556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:15.087346 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:15.087367 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:15.088146 (+ 779us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:15.088687 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:15.088690 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:15.088693 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:15.088914 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:15.089741 (+ 827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:47:15.089751 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:47:15.090252 (+ 501us) client_negotiation.cc:769] Sending connection context
0928 08:47:15.090509 (+ 257us) client_negotiation.cc:240] Negotiation successful
0928 08:47:15.090955 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":325,"threads_started":1}
I20260928 08:47:15.093981 15138 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41033
I20260928 08:47:15.094296 15138 heartbeater.cc:464] Registering TS with master...
I20260928 08:47:15.095242 15138 heartbeater.cc:511] Master 127.0.97.62:41033 requested a full tablet report, sending...
I20260928 08:47:15.097014 14821 ts_manager.cc:194] Registered new tserver with Master: 53c708b11d6142ba939ac6ace881b828 (127.0.97.2:41805)
I20260928 08:47:15.098109 14821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34887
W20260928 08:47:15.426283 15141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:47:15.426730 15141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:47:15.426898 15141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:47:15.434381 15141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:47:15.434511 15141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:47:15.434629 15141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:47:15.445012 15141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:43385
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=33867
--tserver_master_addrs=127.0.97.62:41033
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:47:15.446416 15141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:47:15.447932 15141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:47:15.462366 15147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:15.462466 15146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:15.462433 15149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:47:15.463745 15141 server_base.cc:1034] running on GCE node
I20260928 08:47:15.464568 15141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:47:15.465675 15141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:47:15.466926 15141 hybrid_clock.cc:648] HybridClock initialized: now 1790585235466859 us; error 72 us; skew 500 ppm
I20260928 08:47:15.467352 15141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:47:15 dist-test-slave-n6xs krb5kdc[14236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:47:15.470988 15141 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:47:15.472720 15141 webserver.cc:460] Webserver started at http://127.0.97.3:33867/ using document root <none> and password file <none>
I20260928 08:47:15.473497 15141 fs_manager.cc:359] Metadata directory not provided
I20260928 08:47:15.473664 15141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:47:15.479636 15141 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 08:47:15.483047 15156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:15.484724 15141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:47:15.484925 15141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/wal
uuid: "90ba2aa9a3be471b92e79e27031c753d"
format_stamp: "Formatted at 2026-09-28 08:47:12 on dist-test-slave-n6xs"
server_key: "29149447c5ef103043d690b75402b5a2"
server_key_iv: "bb933ea3b91bf56175b56018d33da89b"
server_key_version: "encryptionkey@0"
I20260928 08:47:15.485967 15141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:47:15.507200 15141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:47:15.512317 15141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:47:15.512639 15141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:47:15.513760 15141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:47:15.515281 15141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:47:15.515394 15141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:15.515561 15141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:47:15.515653 15141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:15.551545 15141 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43385
I20260928 08:47:15.551656 15268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43385 every 8 connection(s)
I20260928 08:47:15.553100 15141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:47:15.559460 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 15141
Sep 28 08:47:15 dist-test-slave-n6xs krb5kdc[14236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:47:15.576865 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:15.560576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38114 (local address 127.0.97.62:41033)
0928 08:47:15.561170 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:15.561178 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:15.561494 (+ 316us) server_negotiation.cc:407] Connection header received
0928 08:47:15.561708 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:15.561712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:15.561770 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:15.561931 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:15.563408 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.564238 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.565594 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.566214 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.566353 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:15.570173 (+ 3820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:15.570181 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:15.570187 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:15.570240 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:15.573730 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:15.574646 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:15.574652 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:15.574657 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:15.574777 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:15.575217 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:15.575221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:15.575223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:15.575517 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:15.575742 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:15.576178 (+ 436us) server_negotiation.cc:299] Negotiation successful
0928 08:47:15.576429 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":347,"threads_started":1}
I20260928 08:47:15.577226 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:15.555969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:54483 (local address 127.0.97.62:41033)
0928 08:47:15.556190 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:15.556196 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:15.557509 (+ 1313us) server_negotiation.cc:407] Connection header received
0928 08:47:15.559418 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:15.559424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:15.559526 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:15.559650 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:15.561854 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.562633 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.564465 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.564903 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.565039 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:15.569373 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:15.569382 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:15.569388 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:15.569436 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:15.573345 (+ 3909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:15.574309 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:15.574315 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:15.574318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:15.574462 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:15.574987 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:15.574991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:15.574993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:15.575358 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:15.575512 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:15.576613 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0928 08:47:15.576811 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 08:47:15.578831 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:15.556584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.3:54483)
0928 08:47:15.557313 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:15.557361 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:15.559090 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:15.559984 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:15.559993 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:15.560814 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:47:15.561643 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:15.561662 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.562822 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.562827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:15.564306 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:15.564318 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.565168 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.565177 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:15.565419 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:15.566177 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:47:15.566206 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:47:15.568968 (+ 2762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:47:15.573556 (+ 4588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:15.573569 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:15.573590 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:15.574165 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:15.574582 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:15.574588 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:15.574594 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:15.574862 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:15.575588 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:47:15.575597 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:47:15.576163 (+ 566us) client_negotiation.cc:769] Sending connection context
0928 08:47:15.576929 (+ 766us) client_negotiation.cc:240] Negotiation successful
0928 08:47:15.577524 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":298,"threads_started":1}
I20260928 08:47:15.580384 15269 heartbeater.cc:347] Connected to a master server at 127.0.97.62:41033
I20260928 08:47:15.580719 15269 heartbeater.cc:464] Registering TS with master...
I20260928 08:47:15.581490 15269 heartbeater.cc:511] Master 127.0.97.62:41033 requested a full tablet report, sending...
I20260928 08:47:15.582971 14821 ts_manager.cc:194] Registered new tserver with Master: 90ba2aa9a3be471b92e79e27031c753d (127.0.97.3:43385)
I20260928 08:47:15.583993 14821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:54483
I20260928 08:47:15.593171 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:47:15.600721 15272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:15.594152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38120 (local address 127.0.97.62:41033)
0928 08:47:15.594402 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:15.594407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:15.594837 (+ 430us) server_negotiation.cc:407] Connection header received
0928 08:47:15.595042 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:15.595046 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:15.595195 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:15.595346 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:47:15.595868 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.596655 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.598268 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.598732 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.598836 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:15.599351 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:47:15.600066 (+ 715us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:47:15.600212 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:15.600407 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 08:47:15.600510 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 08:47:15.601758 14821 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:47:15.609658 14821 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:47:15.613351 15006 heartbeater.cc:503] Master 127.0.97.62:41033 was elected leader, sending a full tablet report...
I20260928 08:47:15.643517 15284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:15.632452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46364 (local address 127.0.97.1:35491)
0928 08:47:15.633916 (+ 1464us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:15.633933 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:15.634130 (+ 197us) server_negotiation.cc:407] Connection header received
0928 08:47:15.634297 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:15.634304 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:15.634532 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:15.634746 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:47:15.635494 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.637454 (+ 1960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.640400 (+ 2946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.641774 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.641922 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:15.642188 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:15.642912 (+ 724us) server_negotiation.cc:299] Negotiation successful
0928 08:47:15.643102 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1242,"thread_start_us":618,"threads_started":1}
I20260928 08:47:15.643567 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:15.632020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35491 (local address 127.0.0.1:46364)
0928 08:47:15.633519 (+ 1499us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:15.633542 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:15.633673 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:15.634746 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:15.634751 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:15.634797 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:47:15.635267 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:15.635278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.637711 (+ 2433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.637716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:15.640236 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:15.640246 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.642435 (+ 2189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.642443 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:15.642651 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:15.642691 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 08:47:15.642870 (+ 179us) client_negotiation.cc:240] Negotiation successful
0928 08:47:15.643102 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1267,"mutex_wait_us":53,"thread_start_us":770,"threads_started":1}
I20260928 08:47:15.644013 15282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:15.626530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47712 (local address 127.0.97.2:41805)
0928 08:47:15.627033 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:15.627040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:15.632555 (+ 5515us) server_negotiation.cc:407] Connection header received
0928 08:47:15.633140 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:15.633153 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:15.633477 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:15.633670 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:47:15.634880 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.636916 (+ 2036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.639832 (+ 2916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.641723 (+ 1891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.641899 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:15.642148 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:15.643372 (+ 1224us) server_negotiation.cc:299] Negotiation successful
0928 08:47:15.643585 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":251,"threads_started":1}
I20260928 08:47:15.644068 15280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:15.625644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57004 (local address 127.0.97.3:43385)
0928 08:47:15.626452 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:15.626459 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:15.631217 (+ 4758us) server_negotiation.cc:407] Connection header received
0928 08:47:15.631580 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:15.631596 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:15.631924 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:15.632170 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:47:15.633875 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.636040 (+ 2165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.640038 (+ 3998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.641723 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.641899 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:15.642199 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:15.643371 (+ 1172us) server_negotiation.cc:299] Negotiation successful
0928 08:47:15.643582 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":444,"threads_started":1}
I20260928 08:47:15.645367 15281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:15.625415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:43385 (local address 127.0.0.1:57004)
0928 08:47:15.630973 (+ 5558us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:15.631027 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:15.631262 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:15.632226 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:15.632237 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:15.632277 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:47:15.632944 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:15.632963 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.636263 (+ 3300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.636268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:15.639785 (+ 3517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:15.639830 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.642492 (+ 2662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.642499 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:15.642936 (+ 437us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:15.643146 (+ 210us) client_negotiation.cc:769] Sending connection context
0928 08:47:15.644358 (+ 1212us) client_negotiation.cc:240] Negotiation successful
0928 08:47:15.644587 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5269,"mutex_wait_us":471,"thread_start_us":6088,"threads_started":1}
I20260928 08:47:15.645416 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:15.625820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41805 (local address 127.0.0.1:47712)
0928 08:47:15.631888 (+ 6068us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:15.631910 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:15.632039 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:15.634127 (+ 2088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:15.634132 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:15.634170 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:47:15.634589 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:15.634604 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.637158 (+ 2554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.637165 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:15.639590 (+ 2425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:15.639605 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.642382 (+ 2777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.642398 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:15.642935 (+ 537us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:15.643144 (+ 209us) client_negotiation.cc:769] Sending connection context
0928 08:47:15.644318 (+ 1174us) client_negotiation.cc:240] Negotiation successful
0928 08:47:15.644646 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5836,"mutex_wait_us":223,"thread_start_us":4664,"threads_started":1}
I20260928 08:47:15.649102 14941 tablet_service.cc:1467] Processing CreateTablet for tablet ccd7d74a09804279bf3eaaed7f09b95d (DEFAULT_TABLE table=test-table [id=7756fa232e0846068ece130e00a61396]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:47:15.649566 15073 tablet_service.cc:1467] Processing CreateTablet for tablet ccd7d74a09804279bf3eaaed7f09b95d (DEFAULT_TABLE table=test-table [id=7756fa232e0846068ece130e00a61396]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:47:15.650071 15204 tablet_service.cc:1467] Processing CreateTablet for tablet ccd7d74a09804279bf3eaaed7f09b95d (DEFAULT_TABLE table=test-table [id=7756fa232e0846068ece130e00a61396]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:47:15.650976 14941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccd7d74a09804279bf3eaaed7f09b95d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:47:15.651149 15073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccd7d74a09804279bf3eaaed7f09b95d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:47:15.651530 15204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccd7d74a09804279bf3eaaed7f09b95d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:47:15.665661 15285 tablet_bootstrap.cc:492] T ccd7d74a09804279bf3eaaed7f09b95d P 90ba2aa9a3be471b92e79e27031c753d: Bootstrap starting.
I20260928 08:47:15.669518 15286 tablet_bootstrap.cc:492] T ccd7d74a09804279bf3eaaed7f09b95d P 53c708b11d6142ba939ac6ace881b828: Bootstrap starting.
I20260928 08:47:15.670962 15285 tablet_bootstrap.cc:654] T ccd7d74a09804279bf3eaaed7f09b95d P 90ba2aa9a3be471b92e79e27031c753d: Neither blocks nor log segments found. Creating new log.
I20260928 08:47:15.671615 15287 tablet_bootstrap.cc:492] T ccd7d74a09804279bf3eaaed7f09b95d P 8604a55c5af4410194362b5517fcf5da: Bootstrap starting.
I20260928 08:47:15.672631 15285 log.cc:824] T ccd7d74a09804279bf3eaaed7f09b95d P 90ba2aa9a3be471b92e79e27031c753d: Log is configured to *not* fsync() on all Append() calls
I20260928 08:47:15.674072 15286 tablet_bootstrap.cc:654] T ccd7d74a09804279bf3eaaed7f09b95d P 53c708b11d6142ba939ac6ace881b828: Neither blocks nor log segments found. Creating new log.
I20260928 08:47:15.675753 15287 tablet_bootstrap.cc:654] T ccd7d74a09804279bf3eaaed7f09b95d P 8604a55c5af4410194362b5517fcf5da: Neither blocks nor log segments found. Creating new log.
I20260928 08:47:15.676024 15286 log.cc:824] T ccd7d74a09804279bf3eaaed7f09b95d P 53c708b11d6142ba939ac6ace881b828: Log is configured to *not* fsync() on all Append() calls
I20260928 08:47:15.676862 15285 tablet_bootstrap.cc:492] T ccd7d74a09804279bf3eaaed7f09b95d P 90ba2aa9a3be471b92e79e27031c753d: No bootstrap required, opened a new log
I20260928 08:47:15.677176 15285 ts_tablet_manager.cc:1397] T ccd7d74a09804279bf3eaaed7f09b95d P 90ba2aa9a3be471b92e79e27031c753d: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260928 08:47:15.677307 15287 log.cc:824] T ccd7d74a09804279bf3eaaed7f09b95d P 8604a55c5af4410194362b5517fcf5da: Log is configured to *not* fsync() on all Append() calls
I20260928 08:47:15.679754 15286 tablet_bootstrap.cc:492] T ccd7d74a09804279bf3eaaed7f09b95d P 53c708b11d6142ba939ac6ace881b828: No bootstrap required, opened a new log
I20260928 08:47:15.680191 15286 ts_tablet_manager.cc:1397] T ccd7d74a09804279bf3eaaed7f09b95d P 53c708b11d6142ba939ac6ace881b828: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.002s
I20260928 08:47:15.680529 15287 tablet_bootstrap.cc:492] T ccd7d74a09804279bf3eaaed7f09b95d P 8604a55c5af4410194362b5517fcf5da: No bootstrap required, opened a new log
I20260928 08:47:15.680882 15287 ts_tablet_manager.cc:1397] T ccd7d74a09804279bf3eaaed7f09b95d P 8604a55c5af4410194362b5517fcf5da: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260928 08:47:15.690006 15285 raft_consensus.cc:348] T ccd7d74a09804279bf3eaaed7f09b95d P 90ba2aa9a3be471b92e79e27031c753d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ba2aa9a3be471b92e79e27031c753d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43385 } } peers { permanent_uuid: "53c708b11d6142ba939ac6ace881b828" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41805 } } peers { permanent_uuid: "8604a55c5af4410194362b5517fcf5da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35491 } }
I20260928 08:47:15.690608 15285 raft_consensus.cc:374] T ccd7d74a09804279bf3eaaed7f09b95d P 90ba2aa9a3be471b92e79e27031c753d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:47:15.690755 15285 raft_consensus.cc:729] T ccd7d74a09804279bf3eaaed7f09b95d P 90ba2aa9a3be471b92e79e27031c753d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90ba2aa9a3be471b92e79e27031c753d, State: Initialized, Role: FOLLOWER
I20260928 08:47:15.690449 15287 raft_consensus.cc:348] T ccd7d74a09804279bf3eaaed7f09b95d P 8604a55c5af4410194362b5517fcf5da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ba2aa9a3be471b92e79e27031c753d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43385 } } peers { permanent_uuid: "53c708b11d6142ba939ac6ace881b828" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41805 } } peers { permanent_uuid: "8604a55c5af4410194362b5517fcf5da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35491 } }
I20260928 08:47:15.690994 15287 raft_consensus.cc:374] T ccd7d74a09804279bf3eaaed7f09b95d P 8604a55c5af4410194362b5517fcf5da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:47:15.691114 15287 raft_consensus.cc:729] T ccd7d74a09804279bf3eaaed7f09b95d P 8604a55c5af4410194362b5517fcf5da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8604a55c5af4410194362b5517fcf5da, State: Initialized, Role: FOLLOWER
I20260928 08:47:15.691516 15285 consensus_queue.cc:260] T ccd7d74a09804279bf3eaaed7f09b95d P 90ba2aa9a3be471b92e79e27031c753d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ba2aa9a3be471b92e79e27031c753d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43385 } } peers { permanent_uuid: "53c708b11d6142ba939ac6ace881b828" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41805 } } peers { permanent_uuid: "8604a55c5af4410194362b5517fcf5da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35491 } }
I20260928 08:47:15.691861 15287 consensus_queue.cc:260] T ccd7d74a09804279bf3eaaed7f09b95d P 8604a55c5af4410194362b5517fcf5da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ba2aa9a3be471b92e79e27031c753d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43385 } } peers { permanent_uuid: "53c708b11d6142ba939ac6ace881b828" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41805 } } peers { permanent_uuid: "8604a55c5af4410194362b5517fcf5da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35491 } }
I20260928 08:47:15.692847 15269 heartbeater.cc:503] Master 127.0.97.62:41033 was elected leader, sending a full tablet report...
I20260928 08:47:15.693018 15286 raft_consensus.cc:348] T ccd7d74a09804279bf3eaaed7f09b95d P 53c708b11d6142ba939ac6ace881b828 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ba2aa9a3be471b92e79e27031c753d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43385 } } peers { permanent_uuid: "53c708b11d6142ba939ac6ace881b828" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41805 } } peers { permanent_uuid: "8604a55c5af4410194362b5517fcf5da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35491 } }
I20260928 08:47:15.693535 15286 raft_consensus.cc:374] T ccd7d74a09804279bf3eaaed7f09b95d P 53c708b11d6142ba939ac6ace881b828 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:47:15.693655 15286 raft_consensus.cc:729] T ccd7d74a09804279bf3eaaed7f09b95d P 53c708b11d6142ba939ac6ace881b828 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53c708b11d6142ba939ac6ace881b828, State: Initialized, Role: FOLLOWER
I20260928 08:47:15.694278 15286 consensus_queue.cc:260] T ccd7d74a09804279bf3eaaed7f09b95d P 53c708b11d6142ba939ac6ace881b828 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ba2aa9a3be471b92e79e27031c753d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43385 } } peers { permanent_uuid: "53c708b11d6142ba939ac6ace881b828" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41805 } } peers { permanent_uuid: "8604a55c5af4410194362b5517fcf5da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35491 } }
I20260928 08:47:15.695125 15285 ts_tablet_manager.cc:1428] T ccd7d74a09804279bf3eaaed7f09b95d P 90ba2aa9a3be471b92e79e27031c753d: Time spent starting tablet: real 0.018s user 0.012s sys 0.004s
I20260928 08:47:15.698709 15138 heartbeater.cc:503] Master 127.0.97.62:41033 was elected leader, sending a full tablet report...
I20260928 08:47:15.701047 15286 ts_tablet_manager.cc:1428] T ccd7d74a09804279bf3eaaed7f09b95d P 53c708b11d6142ba939ac6ace881b828: Time spent starting tablet: real 0.021s user 0.016s sys 0.002s
I20260928 08:47:15.701462 15287 ts_tablet_manager.cc:1428] T ccd7d74a09804279bf3eaaed7f09b95d P 8604a55c5af4410194362b5517fcf5da: Time spent starting tablet: real 0.020s user 0.013s sys 0.007s
I20260928 08:47:15.732973 15291 raft_consensus.cc:482] T ccd7d74a09804279bf3eaaed7f09b95d P 90ba2aa9a3be471b92e79e27031c753d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:47:15.733409 15291 raft_consensus.cc:504] T ccd7d74a09804279bf3eaaed7f09b95d P 90ba2aa9a3be471b92e79e27031c753d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ba2aa9a3be471b92e79e27031c753d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43385 } } peers { permanent_uuid: "53c708b11d6142ba939ac6ace881b828" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41805 } } peers { permanent_uuid: "8604a55c5af4410194362b5517fcf5da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35491 } }
I20260928 08:47:15.735433 15291 leader_election.cc:288] T ccd7d74a09804279bf3eaaed7f09b95d P 90ba2aa9a3be471b92e79e27031c753d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53c708b11d6142ba939ac6ace881b828 (127.0.97.2:41805), 8604a55c5af4410194362b5517fcf5da (127.0.97.1:35491)
I20260928 08:47:15.743499 15282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:15.736089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:40901 (local address 127.0.97.2:41805)
0928 08:47:15.736317 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:15.736325 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:15.736525 (+ 200us) server_negotiation.cc:407] Connection header received
0928 08:47:15.736809 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:15.736816 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:15.736911 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:15.737029 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:47:15.737523 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.738766 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.741266 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.742559 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.742729 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:15.742852 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:15.743092 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 08:47:15.743235 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 08:47:15.743526 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:15.735898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:41805 (local address 127.0.97.3:40901)
0928 08:47:15.736338 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:15.736366 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:15.736602 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:15.737077 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:15.737083 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:15.737124 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:47:15.737341 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:15.737356 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.738938 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.738942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:15.741070 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:15.741086 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.742764 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.742769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:15.742914 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:15.742941 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 08:47:15.743022 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 08:47:15.743210 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254}
I20260928 08:47:15.744031 15284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:15.736002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:34363 (local address 127.0.97.1:35491)
0928 08:47:15.736240 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:15.736250 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:15.737057 (+ 807us) server_negotiation.cc:407] Connection header received
0928 08:47:15.737225 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:15.737231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:15.737311 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:15.737421 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:47:15.738198 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.739196 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.741567 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.742897 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.743122 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:15.743220 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:15.743557 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 08:47:15.743730 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 08:47:15.744086 15294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:15.735998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35491 (local address 127.0.97.3:34363)
0928 08:47:15.736896 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:15.736918 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:15.737072 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:15.737489 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:15.737495 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:15.737520 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:47:15.737996 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:15.738009 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.739336 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.739342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:15.741389 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:15.741400 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.743192 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.743198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:15.743360 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:15.743408 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 08:47:15.743502 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 08:47:15.743653 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"mutex_wait_us":1,"thread_start_us":466,"threads_started":1}
I20260928 08:47:15.744964 15093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccd7d74a09804279bf3eaaed7f09b95d" candidate_uuid: "90ba2aa9a3be471b92e79e27031c753d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53c708b11d6142ba939ac6ace881b828" is_pre_election: true
I20260928 08:47:15.745103 14961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccd7d74a09804279bf3eaaed7f09b95d" candidate_uuid: "90ba2aa9a3be471b92e79e27031c753d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8604a55c5af4410194362b5517fcf5da" is_pre_election: true
I20260928 08:47:15.745522 15093 raft_consensus.cc:2454] T ccd7d74a09804279bf3eaaed7f09b95d P 53c708b11d6142ba939ac6ace881b828 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90ba2aa9a3be471b92e79e27031c753d in term 0.
I20260928 08:47:15.745694 14961 raft_consensus.cc:2454] T ccd7d74a09804279bf3eaaed7f09b95d P 8604a55c5af4410194362b5517fcf5da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90ba2aa9a3be471b92e79e27031c753d in term 0.
I20260928 08:47:15.746362 15159 leader_election.cc:302] T ccd7d74a09804279bf3eaaed7f09b95d P 90ba2aa9a3be471b92e79e27031c753d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53c708b11d6142ba939ac6ace881b828, 90ba2aa9a3be471b92e79e27031c753d; no voters:
I20260928 08:47:15.746825 15291 raft_consensus.cc:2781] T ccd7d74a09804279bf3eaaed7f09b95d P 90ba2aa9a3be471b92e79e27031c753d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:47:15.747001 15291 raft_consensus.cc:482] T ccd7d74a09804279bf3eaaed7f09b95d P 90ba2aa9a3be471b92e79e27031c753d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:47:15.747134 15291 raft_consensus.cc:3037] T ccd7d74a09804279bf3eaaed7f09b95d P 90ba2aa9a3be471b92e79e27031c753d [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:47:15.749634 15291 raft_consensus.cc:504] T ccd7d74a09804279bf3eaaed7f09b95d P 90ba2aa9a3be471b92e79e27031c753d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ba2aa9a3be471b92e79e27031c753d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43385 } } peers { permanent_uuid: "53c708b11d6142ba939ac6ace881b828" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41805 } } peers { permanent_uuid: "8604a55c5af4410194362b5517fcf5da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35491 } }
I20260928 08:47:15.750341 15291 leader_election.cc:288] T ccd7d74a09804279bf3eaaed7f09b95d P 90ba2aa9a3be471b92e79e27031c753d [CANDIDATE]: Term 1 election: Requested vote from peers 53c708b11d6142ba939ac6ace881b828 (127.0.97.2:41805), 8604a55c5af4410194362b5517fcf5da (127.0.97.1:35491)
I20260928 08:47:15.750949 15093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccd7d74a09804279bf3eaaed7f09b95d" candidate_uuid: "90ba2aa9a3be471b92e79e27031c753d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53c708b11d6142ba939ac6ace881b828"
I20260928 08:47:15.751020 14961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccd7d74a09804279bf3eaaed7f09b95d" candidate_uuid: "90ba2aa9a3be471b92e79e27031c753d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8604a55c5af4410194362b5517fcf5da"
I20260928 08:47:15.751186 15093 raft_consensus.cc:3037] T ccd7d74a09804279bf3eaaed7f09b95d P 53c708b11d6142ba939ac6ace881b828 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:47:15.751313 14961 raft_consensus.cc:3037] T ccd7d74a09804279bf3eaaed7f09b95d P 8604a55c5af4410194362b5517fcf5da [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:47:15.753742 14961 raft_consensus.cc:2454] T ccd7d74a09804279bf3eaaed7f09b95d P 8604a55c5af4410194362b5517fcf5da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90ba2aa9a3be471b92e79e27031c753d in term 1.
I20260928 08:47:15.753794 15093 raft_consensus.cc:2454] T ccd7d74a09804279bf3eaaed7f09b95d P 53c708b11d6142ba939ac6ace881b828 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90ba2aa9a3be471b92e79e27031c753d in term 1.
I20260928 08:47:15.754298 15159 leader_election.cc:302] T ccd7d74a09804279bf3eaaed7f09b95d P 90ba2aa9a3be471b92e79e27031c753d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53c708b11d6142ba939ac6ace881b828, 90ba2aa9a3be471b92e79e27031c753d; no voters:
I20260928 08:47:15.754664 15291 raft_consensus.cc:2781] T ccd7d74a09804279bf3eaaed7f09b95d P 90ba2aa9a3be471b92e79e27031c753d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:47:15.755151 15291 raft_consensus.cc:686] T ccd7d74a09804279bf3eaaed7f09b95d P 90ba2aa9a3be471b92e79e27031c753d [term 1 LEADER]: Becoming Leader. State: Replica: 90ba2aa9a3be471b92e79e27031c753d, State: Running, Role: LEADER
I20260928 08:47:15.755662 15291 consensus_queue.cc:237] T ccd7d74a09804279bf3eaaed7f09b95d P 90ba2aa9a3be471b92e79e27031c753d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ba2aa9a3be471b92e79e27031c753d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43385 } } peers { permanent_uuid: "53c708b11d6142ba939ac6ace881b828" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41805 } } peers { permanent_uuid: "8604a55c5af4410194362b5517fcf5da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35491 } }
I20260928 08:47:15.761065 14821 catalog_manager.cc:5495] T ccd7d74a09804279bf3eaaed7f09b95d P 90ba2aa9a3be471b92e79e27031c753d reported cstate change: term changed from 0 to 1, leader changed from <none> to 90ba2aa9a3be471b92e79e27031c753d (127.0.97.3). New cstate: current_term: 1 leader_uuid: "90ba2aa9a3be471b92e79e27031c753d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90ba2aa9a3be471b92e79e27031c753d" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 43385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53c708b11d6142ba939ac6ace881b828" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8604a55c5af4410194362b5517fcf5da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35491 } health_report { overall_health: UNKNOWN } } }
I20260928 08:47:15.785526 15280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:15.779679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57018 (local address 127.0.97.3:43385)
0928 08:47:15.779985 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:15.779991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:15.780063 (+ 72us) server_negotiation.cc:407] Connection header received
0928 08:47:15.780321 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:15.780326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:15.780407 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:15.780577 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:47:15.780978 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.781697 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.783180 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.783604 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.783742 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:15.784332 (+ 590us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:47:15.784889 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:47:15.785064 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:15.785116 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 08:47:15.785206 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260928 08:47:15.795754 14961 raft_consensus.cc:1261] T ccd7d74a09804279bf3eaaed7f09b95d P 8604a55c5af4410194362b5517fcf5da [term 1 FOLLOWER]: Refusing update from remote peer 90ba2aa9a3be471b92e79e27031c753d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:47:15.796100 15093 raft_consensus.cc:1261] T ccd7d74a09804279bf3eaaed7f09b95d P 53c708b11d6142ba939ac6ace881b828 [term 1 FOLLOWER]: Refusing update from remote peer 90ba2aa9a3be471b92e79e27031c753d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:47:15.797122 15295 consensus_queue.cc:1035] T ccd7d74a09804279bf3eaaed7f09b95d P 90ba2aa9a3be471b92e79e27031c753d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8604a55c5af4410194362b5517fcf5da" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 35491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:47:15.797559 15291 consensus_queue.cc:1035] T ccd7d74a09804279bf3eaaed7f09b95d P 90ba2aa9a3be471b92e79e27031c753d [LEADER]: Connected to new peer: Peer: permanent_uuid: "53c708b11d6142ba939ac6ace881b828" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 41805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 08:47:15.804795 15270 tablet.cc:2369] T ccd7d74a09804279bf3eaaed7f09b95d P 90ba2aa9a3be471b92e79e27031c753d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:47:15.828111 14821 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:38120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:47:15.828501 14821 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:38120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:47:15.833142 14821 catalog_manager.cc:5782] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5: Sending DeleteTablet for 3 replicas of tablet ccd7d74a09804279bf3eaaed7f09b95d
I20260928 08:47:15.834738 14941 tablet_service.cc:1514] Processing DeleteTablet for tablet ccd7d74a09804279bf3eaaed7f09b95d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:47:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:46364
I20260928 08:47:15.834738 15073 tablet_service.cc:1514] Processing DeleteTablet for tablet ccd7d74a09804279bf3eaaed7f09b95d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:47:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47712
I20260928 08:47:15.834924 15204 tablet_service.cc:1514] Processing DeleteTablet for tablet ccd7d74a09804279bf3eaaed7f09b95d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:47:15 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57004
I20260928 08:47:15.836442 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 14858
I20260928 08:47:15.848166 15307 tablet_replica.cc:330] T ccd7d74a09804279bf3eaaed7f09b95d P 8604a55c5af4410194362b5517fcf5da: stopping tablet replica
I20260928 08:47:15.848801 15307 raft_consensus.cc:2229] T ccd7d74a09804279bf3eaaed7f09b95d P 8604a55c5af4410194362b5517fcf5da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:47:15.849532 15307 raft_consensus.cc:2258] T ccd7d74a09804279bf3eaaed7f09b95d P 8604a55c5af4410194362b5517fcf5da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:47:15.852432 15307 ts_tablet_manager.cc:1905] T ccd7d74a09804279bf3eaaed7f09b95d P 8604a55c5af4410194362b5517fcf5da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:47:15.856438 15308 tablet_replica.cc:330] T ccd7d74a09804279bf3eaaed7f09b95d P 53c708b11d6142ba939ac6ace881b828: stopping tablet replica
I20260928 08:47:15.857035 15308 raft_consensus.cc:2229] T ccd7d74a09804279bf3eaaed7f09b95d P 53c708b11d6142ba939ac6ace881b828 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:47:15.857774 15308 raft_consensus.cc:2258] T ccd7d74a09804279bf3eaaed7f09b95d P 53c708b11d6142ba939ac6ace881b828 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:47:15.857923 15309 tablet_replica.cc:330] T ccd7d74a09804279bf3eaaed7f09b95d P 90ba2aa9a3be471b92e79e27031c753d: stopping tablet replica
I20260928 08:47:15.858803 15309 raft_consensus.cc:2229] T ccd7d74a09804279bf3eaaed7f09b95d P 90ba2aa9a3be471b92e79e27031c753d [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:47:15.860602 15284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:15.837516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46370 (local address 127.0.97.1:35491)
0928 08:47:15.837719 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:15.837729 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:15.837775 (+ 46us) server_negotiation.cc:407] Connection header received
0928 08:47:15.838062 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:15.838069 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:15.838214 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:15.838358 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:15.841259 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.842695 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.846003 (+ 3308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.846876 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.847058 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:15.851142 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:15.851165 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:15.851187 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:15.851248 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:15.856122 (+ 4874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:15.857664 (+ 1542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:15.857678 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:15.857682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:15.857883 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:15.858458 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:15.858465 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:15.858469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:15.859050 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:15.859418 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:15.859716 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 08:47:15.860097 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 08:47:15.861022 15309 raft_consensus.cc:2258] T ccd7d74a09804279bf3eaaed7f09b95d P 90ba2aa9a3be471b92e79e27031c753d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:47:15.861590 15001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:47:15.863029 15308 ts_tablet_manager.cc:1905] T ccd7d74a09804279bf3eaaed7f09b95d P 53c708b11d6142ba939ac6ace881b828: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:47:15.864483 15309 ts_tablet_manager.cc:1905] T ccd7d74a09804279bf3eaaed7f09b95d P 90ba2aa9a3be471b92e79e27031c753d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:47:15.871536 15308 ts_tablet_manager.cc:1918] T ccd7d74a09804279bf3eaaed7f09b95d P 53c708b11d6142ba939ac6ace881b828: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:47:15.871889 15308 log.cc:1196] T ccd7d74a09804279bf3eaaed7f09b95d P 53c708b11d6142ba939ac6ace881b828: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/wal/wals/ccd7d74a09804279bf3eaaed7f09b95d
I20260928 08:47:15.872658 15308 ts_tablet_manager.cc:1939] T ccd7d74a09804279bf3eaaed7f09b95d P 53c708b11d6142ba939ac6ace881b828: Deleting consensus metadata
I20260928 08:47:15.875015 15309 ts_tablet_manager.cc:1918] T ccd7d74a09804279bf3eaaed7f09b95d P 90ba2aa9a3be471b92e79e27031c753d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:47:15.875175 14807 catalog_manager.cc:4841] TS 53c708b11d6142ba939ac6ace881b828 (127.0.97.2:41805): tablet ccd7d74a09804279bf3eaaed7f09b95d (table test-table [id=7756fa232e0846068ece130e00a61396]) successfully deleted
I20260928 08:47:15.875312 15309 log.cc:1196] T ccd7d74a09804279bf3eaaed7f09b95d P 90ba2aa9a3be471b92e79e27031c753d: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/wal/wals/ccd7d74a09804279bf3eaaed7f09b95d
I20260928 08:47:15.876180 15309 ts_tablet_manager.cc:1939] T ccd7d74a09804279bf3eaaed7f09b95d P 90ba2aa9a3be471b92e79e27031c753d: Deleting consensus metadata
I20260928 08:47:15.879091 14805 catalog_manager.cc:4841] TS 90ba2aa9a3be471b92e79e27031c753d (127.0.97.3:43385): tablet ccd7d74a09804279bf3eaaed7f09b95d (table test-table [id=7756fa232e0846068ece130e00a61396]) successfully deleted
I20260928 08:47:15.967913 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 14858
I20260928 08:47:15.987766 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 15010
W20260928 08:47:15.987769 14805 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:35491 (error 108)
I20260928 08:47:15.990379 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:15.989799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35491 (local address 127.0.0.1:46374)
0928 08:47:15.990020 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:15.990167 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260928 08:47:15.990716 14805 catalog_manager.cc:4584] TS 8604a55c5af4410194362b5517fcf5da (127.0.97.1:35491): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ccd7d74a09804279bf3eaaed7f09b95d: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35491: connect: Connection refused (error 111)
I20260928 08:47:16.005236 15282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:15.989043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47728 (local address 127.0.97.2:41805)
0928 08:47:15.989258 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:15.989265 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:15.989282 (+ 17us) server_negotiation.cc:407] Connection header received
0928 08:47:15.989346 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:15.989350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:15.989417 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:15.989548 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:15.991621 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.992665 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.994220 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:15.994663 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:15.994836 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:15.998109 (+ 3273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:15.998130 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:15.998146 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:15.998205 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:16.001598 (+ 3393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:16.002590 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:16.002599 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:16.002602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:16.002767 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:16.003258 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:16.003264 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:16.003275 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:16.003723 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:16.004148 (+ 425us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:16.004528 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 08:47:16.004790 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 08:47:16.006059 15133 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:47:16.045043 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:16.044432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35491 (local address 127.0.0.1:46380)
0928 08:47:16.044696 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:16.044827 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260928 08:47:16.046247 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:16.045846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35491 (local address 127.0.0.1:46382)
0928 08:47:16.046011 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:16.046110 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 08:47:16.099747 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 15010
I20260928 08:47:16.106986 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:16.106320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35491 (local address 127.0.0.1:46398)
0928 08:47:16.106613 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:16.106738 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260928 08:47:16.108325 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:16.107878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35491 (local address 127.0.0.1:46402)
0928 08:47:16.108059 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:16.108160 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 08:47:16.119458 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 15141
I20260928 08:47:16.139663 15280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:16.120294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57028 (local address 127.0.97.3:43385)
0928 08:47:16.120518 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:16.120529 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:16.120581 (+ 52us) server_negotiation.cc:407] Connection header received
0928 08:47:16.120812 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:16.120824 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:16.120904 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:16.121028 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:16.122417 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:16.123309 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:16.124753 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:16.125283 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:16.125451 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:16.132569 (+ 7118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:16.132585 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:16.132604 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:16.132657 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:16.136306 (+ 3649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:16.137224 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:16.137232 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:16.137235 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:16.137428 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:16.137887 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:16.137891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:16.137893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:16.138308 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:16.138678 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:16.139009 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 08:47:16.139260 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260928 08:47:16.140501 15264 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:47:16.189419 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:16.188817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35491 (local address 127.0.0.1:46406)
0928 08:47:16.189135 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:16.189234 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260928 08:47:16.190666 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:16.190201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35491 (local address 127.0.0.1:46420)
0928 08:47:16.190464 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:16.190546 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260928 08:47:16.266644 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 15141
I20260928 08:47:16.286741 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 14788
I20260928 08:47:16.288185 14851 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:47:16.403121 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:16.402655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:35491 (local address 127.0.0.1:46428)
0928 08:47:16.402811 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:16.402894 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 08:47:16.403162 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 14788
I20260928 08:47:16.424844 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41033
--webserver_interface=127.0.97.62
--webserver_port=42203
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:41033
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:47:16.770402 15315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:47:16.770748 15315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:47:16.770969 15315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:47:16.778764 15315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:47:16.778939 15315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:47:16.779063 15315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:47:16.779181 15315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:47:16.779250 15315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:47:16.789688 15315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:41033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:41033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=42203
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:47:16.791199 15315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:47:16.792650 15315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:47:16.805994 15320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:16.806039 15323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:16.805964 15321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:47:16.807441 15315 server_base.cc:1034] running on GCE node
I20260928 08:47:16.808512 15315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:47:16.810309 15315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:47:16.811575 15315 hybrid_clock.cc:648] HybridClock initialized: now 1790585236811519 us; error 55 us; skew 500 ppm
I20260928 08:47:16.812116 15315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:47:16 dist-test-slave-n6xs krb5kdc[14236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:47:16.815671 15315 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:47:16.817559 15315 webserver.cc:460] Webserver started at http://127.0.97.62:42203/ using document root <none> and password file <none>
I20260928 08:47:16.818372 15315 fs_manager.cc:359] Metadata directory not provided
I20260928 08:47:16.818540 15315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:47:16.824996 15315 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 08:47:16.828574 15330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:16.830207 15315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:47:16.830461 15315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "aa24014e8c13464f8070bdccea5f1dd5"
format_stamp: "Formatted at 2026-09-28 08:47:11 on dist-test-slave-n6xs"
server_key: "1d6526fab231a8c99f1fca0056a36a8b"
server_key_iv: "5461d6bc2d753e9db4be248c2d951949"
server_key_version: "encryptionkey@0"
I20260928 08:47:16.831560 15315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:47:16.846267 15315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:47:16.851123 15315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:47:16.851511 15315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:47:16.871623 15315 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:41033
I20260928 08:47:16.871646 15381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:41033 every 8 connection(s)
I20260928 08:47:16.873356 15315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:47:16.878860 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 15315
I20260928 08:47:16.880031 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:35491
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:41033
--webserver_port=34731
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:47:16.881359 15382 sys_catalog.cc:263] Verifying existing consensus state
I20260928 08:47:16.885105 15382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5: Bootstrap starting.
I20260928 08:47:16.917083 15382 log.cc:824] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5: Log is configured to *not* fsync() on all Append() calls
I20260928 08:47:16.938062 15382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:47:16.938721 15382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5: Bootstrap complete.
I20260928 08:47:16.948726 15382 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa24014e8c13464f8070bdccea5f1dd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41033 } }
I20260928 08:47:16.949854 15382 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa24014e8c13464f8070bdccea5f1dd5, State: Initialized, Role: FOLLOWER
I20260928 08:47:16.950615 15382 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa24014e8c13464f8070bdccea5f1dd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41033 } }
I20260928 08:47:16.950899 15382 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:47:16.951040 15382 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:47:16.951244 15382 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [term 2 FOLLOWER]: Advancing to term 3
I20260928 08:47:16.955488 15382 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa24014e8c13464f8070bdccea5f1dd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41033 } }
I20260928 08:47:16.956178 15382 leader_election.cc:302] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa24014e8c13464f8070bdccea5f1dd5; no voters:
I20260928 08:47:16.956882 15382 leader_election.cc:288] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 08:47:16.957110 15386 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 08:47:16.958346 15386 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [term 3 LEADER]: Becoming Leader. State: Replica: aa24014e8c13464f8070bdccea5f1dd5, State: Running, Role: LEADER
I20260928 08:47:16.959231 15382 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:47:16.959076 15386 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa24014e8c13464f8070bdccea5f1dd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41033 } }
I20260928 08:47:16.962040 15388 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa24014e8c13464f8070bdccea5f1dd5. Latest consensus state: current_term: 3 leader_uuid: "aa24014e8c13464f8070bdccea5f1dd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa24014e8c13464f8070bdccea5f1dd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41033 } } }
I20260928 08:47:16.962970 15388 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [sys.catalog]: This master's current role is: LEADER
I20260928 08:47:16.962612 15387 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "aa24014e8c13464f8070bdccea5f1dd5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa24014e8c13464f8070bdccea5f1dd5" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 41033 } } }
I20260928 08:47:16.963572 15387 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa24014e8c13464f8070bdccea5f1dd5 [sys.catalog]: This master's current role is: LEADER
I20260928 08:47:16.973285 15393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:47:16.985653 15393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:47:16.987970 15393 catalog_manager.cc:1259] Loaded cluster ID: 3fd605c65cc848b98489e0243a4b9e58
I20260928 08:47:16.988238 15393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 08:47:16.991446 15393 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f0b8467b9ad google::LogMessage::Fail() at ??:0
@ 0x7f0b8467e88a google::LogMessage::SendToLog() at ??:0
@ 0x7f0b8467b4de google::LogMessage::Flush() at ??:0
@ 0x7f0b8467ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0b9a0ae512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f0b9a0ab976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0b9a133610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f0b9a133439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0b872c7345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0b872ca91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f0b872ca799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0b872a5eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0b8a57f6db start_thread at ??:0
@ 0x7f0b8351561f clone at ??:0
W20260928 08:47:17.311138 15384 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:47:17.311518 15384 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:47:17.311694 15384 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:47:17.319370 15384 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:47:17.319569 15384 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:47:17.319763 15384 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:47:17.330323 15384 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:35491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=34731
--tserver_master_addrs=127.0.97.62:41033
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:47:17.331792 15384 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:47:17.333310 15384 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:47:17.347693 15423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:17.347774 15426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:17.347743 15424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:47:17.348798 15384 server_base.cc:1034] running on GCE node
I20260928 08:47:17.349426 15384 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:47:17.350449 15384 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:47:17.351667 15384 hybrid_clock.cc:648] HybridClock initialized: now 1790585237351584 us; error 73 us; skew 500 ppm
I20260928 08:47:17.352075 15384 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:47:17 dist-test-slave-n6xs krb5kdc[14236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:47:17.355433 15384 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:47:17.357080 15384 webserver.cc:460] Webserver started at http://127.0.97.1:34731/ using document root <none> and password file <none>
I20260928 08:47:17.357808 15384 fs_manager.cc:359] Metadata directory not provided
I20260928 08:47:17.357925 15384 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:47:17.364300 15384 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 08:47:17.370007 15433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:17.371698 15384 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260928 08:47:17.371935 15384 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/wal
uuid: "8604a55c5af4410194362b5517fcf5da"
format_stamp: "Formatted at 2026-09-28 08:47:11 on dist-test-slave-n6xs"
server_key: "8ae245d09be6f937dc3b85b1db1e7425"
server_key_iv: "baa2e661e39fb911be6fcdb1261eab03"
server_key_version: "encryptionkey@0"
I20260928 08:47:17.372970 15384 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:47:17.393288 15384 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:47:17.398103 15384 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:47:17.398346 15384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:47:17.399322 15384 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:47:17.401270 15440 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260928 08:47:17.407512 15440 ts_tablet_manager.cc:1852] T ccd7d74a09804279bf3eaaed7f09b95d P 8604a55c5af4410194362b5517fcf5da: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 08:47:17.408697 15440 ts_tablet_manager.cc:1905] T ccd7d74a09804279bf3eaaed7f09b95d P 8604a55c5af4410194362b5517fcf5da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:47:17.421424 15440 ts_tablet_manager.cc:1918] T ccd7d74a09804279bf3eaaed7f09b95d P 8604a55c5af4410194362b5517fcf5da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:47:17.421705 15440 log.cc:1196] T ccd7d74a09804279bf3eaaed7f09b95d P 8604a55c5af4410194362b5517fcf5da: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/wal/wals/ccd7d74a09804279bf3eaaed7f09b95d
I20260928 08:47:17.422335 15440 ts_tablet_manager.cc:1939] T ccd7d74a09804279bf3eaaed7f09b95d P 8604a55c5af4410194362b5517fcf5da: Deleting consensus metadata
I20260928 08:47:17.422840 15384 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 08:47:17.422971 15384 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.022s user 0.001s sys 0.000s
I20260928 08:47:17.423071 15384 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:47:17.423136 15384 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:17.459905 15384 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:35491
I20260928 08:47:17.459926 15546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:35491 every 8 connection(s)
I20260928 08:47:17.461374 15384 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:47:17.465112 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 15384
I20260928 08:47:17.465785 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:41805
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:41033
--webserver_port=40595
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:47:17.466862 15549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:17.464690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.1:35793)
0928 08:47:17.465582 (+ 892us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:17.465859 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":358,"threads_started":1}
W20260928 08:47:17.467623 15547 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41033 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:41033: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
I20260928 08:47:17.468603 15549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:17.468227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.1:54149)
0928 08:47:17.468396 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:17.468479 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 08:47:17.469604 15549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:17.469214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.1:46157)
0928 08:47:17.469383 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:17.469469 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260928 08:47:17.469864 15547 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 08:47:17.816727 15550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:47:17.817124 15550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:47:17.817309 15550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:47:17.824946 15550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:47:17.825138 15550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:47:17.825344 15550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:47:17.835700 15550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:41805
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=40595
--tserver_master_addrs=127.0.97.62:41033
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:47:17.837169 15550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:47:17.838641 15550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:47:17.853446 15556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:17.853446 15555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:17.853478 15558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:47:17.854279 15550 server_base.cc:1034] running on GCE node
I20260928 08:47:17.855109 15550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:47:17.856197 15550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:47:17.857479 15550 hybrid_clock.cc:648] HybridClock initialized: now 1790585237857403 us; error 68 us; skew 500 ppm
I20260928 08:47:17.857862 15550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:47:17 dist-test-slave-n6xs krb5kdc[14236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:47:17.861459 15550 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:47:17.863160 15550 webserver.cc:460] Webserver started at http://127.0.97.2:40595/ using document root <none> and password file <none>
I20260928 08:47:17.863977 15550 fs_manager.cc:359] Metadata directory not provided
I20260928 08:47:17.864130 15550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:47:17.870062 15550 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 08:47:17.873281 15565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:17.874886 15550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:47:17.875067 15550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/wal
uuid: "53c708b11d6142ba939ac6ace881b828"
format_stamp: "Formatted at 2026-09-28 08:47:12 on dist-test-slave-n6xs"
server_key: "f4b72c65f9705c2d14dd9b775a1a442f"
server_key_iv: "0aeadb982c5c08c7540275da145ca85c"
server_key_version: "encryptionkey@0"
I20260928 08:47:17.876242 15550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:47:17.892665 15550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:47:17.897380 15550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:47:17.897704 15550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:47:17.898806 15550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:47:17.900357 15550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:47:17.900447 15550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:17.900616 15550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:47:17.900691 15550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:17.935426 15550 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:41805
I20260928 08:47:17.935528 15677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:41805 every 8 connection(s)
I20260928 08:47:17.937132 15550 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:47:17.941533 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 15550
I20260928 08:47:17.942277 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:43385
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:41033
--webserver_port=33867
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:35997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:47:17.942929 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:17.940606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.2:36793)
0928 08:47:17.941446 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:17.941689 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":359,"threads_started":1}
W20260928 08:47:17.943722 15678 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41033 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:41033: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
I20260928 08:47:17.944727 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:17.944388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.2:47391)
0928 08:47:17.944547 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:17.944620 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 08:47:17.945689 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:17.945382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.2:39653)
0928 08:47:17.945514 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:17.945585 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 08:47:17.946002 15678 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 08:47:18.295568 15681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:47:18.296028 15681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:47:18.296211 15681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:47:18.303573 15681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:47:18.303725 15681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:47:18.303977 15681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:47:18.314117 15681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:43385
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=33867
--tserver_master_addrs=127.0.97.62:41033
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:47:18.315462 15681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:47:18.316840 15681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:47:18.330206 15687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:18.330134 15689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:18.330633 15686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:47:18.331493 15681 server_base.cc:1034] running on GCE node
I20260928 08:47:18.332181 15681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:47:18.333173 15681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:47:18.334391 15681 hybrid_clock.cc:648] HybridClock initialized: now 1790585238334333 us; error 44 us; skew 500 ppm
I20260928 08:47:18.334760 15681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:47:18 dist-test-slave-n6xs krb5kdc[14236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:47:18.337951 15681 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:47:18.339315 15681 webserver.cc:460] Webserver started at http://127.0.97.3:33867/ using document root <none> and password file <none>
I20260928 08:47:18.340078 15681 fs_manager.cc:359] Metadata directory not provided
I20260928 08:47:18.340180 15681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:47:18.345446 15681 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:47:18.348171 15696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:18.349531 15681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:47:18.349702 15681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/wal
uuid: "90ba2aa9a3be471b92e79e27031c753d"
format_stamp: "Formatted at 2026-09-28 08:47:12 on dist-test-slave-n6xs"
server_key: "29149447c5ef103043d690b75402b5a2"
server_key_iv: "bb933ea3b91bf56175b56018d33da89b"
server_key_version: "encryptionkey@0"
I20260928 08:47:18.350690 15681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:47:18.370793 15681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:47:18.375392 15681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:47:18.375764 15681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:47:18.376924 15681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:47:18.378376 15681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:47:18.378496 15681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:18.378614 15681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:47:18.378692 15681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:18.412344 15681 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:43385
I20260928 08:47:18.412395 15808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:43385 every 8 connection(s)
I20260928 08:47:18.413805 15681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:47:18.416651 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 15681
I20260928 08:47:18.418879 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:18.416823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.3:48373)
0928 08:47:18.417637 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:18.417872 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":346,"threads_started":1}
W20260928 08:47:18.419561 15809 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:41033 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:41033: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
I20260928 08:47:18.420298 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:18.419997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.3:51265)
0928 08:47:18.420143 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:18.420201 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 08:47:18.421113 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:18.420777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.3:53373)
0928 08:47:18.420910 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:18.421017 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 08:47:18.421329 15809 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 08:47:18.471264 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:18.470440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.1:49309)
0928 08:47:18.470935 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:18.471022 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":225,"threads_started":1}
I20260928 08:47:18.947440 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:18.946637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.2:60193)
0928 08:47:18.947120 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:18.947207 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":247,"threads_started":1}
I20260928 08:47:19.422894 15815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:19.421981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.3:53463)
0928 08:47:19.422473 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:19.422574 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":231,"threads_started":1}
I20260928 08:47:19.473061 15816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:19.472348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.1:57713)
0928 08:47:19.472814 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:19.472901 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":239,"threads_started":1}
I20260928 08:47:19.949353 15817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:19.948512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.2:34671)
0928 08:47:19.949042 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:19.949154 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":252,"threads_started":1}
I20260928 08:47:20.424531 15818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:20.423753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.3:48887)
0928 08:47:20.424255 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:20.424347 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":264,"threads_started":1}
I20260928 08:47:20.474864 15819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:20.474125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.1:50103)
0928 08:47:20.474598 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:20.474685 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":236,"threads_started":1}
I20260928 08:47:20.951237 15820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:20.950465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.2:34961)
0928 08:47:20.950984 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:20.951063 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":332,"threads_started":1}
I20260928 08:47:21.426332 15821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:21.425504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.3:53871)
0928 08:47:21.426029 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:21.426144 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":314,"threads_started":1}
I20260928 08:47:21.476809 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:21.476001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.1:41457)
0928 08:47:21.476465 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:21.476585 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":240,"threads_started":1}
I20260928 08:47:21.952852 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:21.952147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.2:51705)
0928 08:47:21.952577 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:21.952656 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":198,"threads_started":1}
I20260928 08:47:22.428124 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:22.427353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.3:36939)
0928 08:47:22.427795 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:22.427932 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":201,"threads_started":1}
I20260928 08:47:22.478360 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:22.477649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.1:58917)
0928 08:47:22.478083 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:22.478180 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":219,"threads_started":1}
I20260928 08:47:22.954849 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:22.953914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.2:50037)
0928 08:47:22.954474 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:22.954572 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":262,"threads_started":1}
I20260928 08:47:23.429632 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:23.428948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.3:42669)
0928 08:47:23.429391 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:23.429468 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":215,"threads_started":1}
I20260928 08:47:23.480131 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:23.479254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.1:45031)
0928 08:47:23.479796 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:23.479924 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":275,"threads_started":1}
I20260928 08:47:23.956405 15829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:23.955680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.2:51467)
0928 08:47:23.956156 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:23.956241 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":273,"threads_started":1}
I20260928 08:47:24.431321 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:24.430518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.3:52415)
0928 08:47:24.431075 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:24.431161 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":330,"threads_started":1}
I20260928 08:47:24.481849 15831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:24.481051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.1:43479)
0928 08:47:24.481569 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:24.481647 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":264,"threads_started":1}
I20260928 08:47:24.958209 15832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:24.957346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.2:55477)
0928 08:47:24.957890 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:24.957970 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":272,"threads_started":1}
I20260928 08:47:25.433264 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:25.432265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.3:47809)
0928 08:47:25.432962 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:25.433045 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":389,"threads_started":1}
I20260928 08:47:25.483415 15834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:25.482729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.1:36901)
0928 08:47:25.483170 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:25.483250 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":248,"threads_started":1}
I20260928 08:47:25.960233 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:25.959381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.2:55987)
0928 08:47:25.959980 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:25.960067 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":293,"threads_started":1}
I20260928 08:47:26.435163 15836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:26.434331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.3:42131)
0928 08:47:26.434894 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:26.434992 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":330,"threads_started":1}
I20260928 08:47:26.485324 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:26.484481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.1:52467)
0928 08:47:26.485024 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:26.485141 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":279,"threads_started":1}
I20260928 08:47:26.962138 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:26.961237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.2:41077)
0928 08:47:26.961858 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:26.961951 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":327,"threads_started":1}
I20260928 08:47:27.437510 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:27.436259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.3:35003)
0928 08:47:27.437183 (+ 924us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:27.437268 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":273,"threads_started":1}
I20260928 08:47:27.487006 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:27.486326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.1:56853)
0928 08:47:27.486751 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:27.486834 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":215,"threads_started":1}
I20260928 08:47:27.964041 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:27.963132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.2:53727)
0928 08:47:27.963700 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:27.963837 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":263,"threads_started":1}
I20260928 08:47:28.439527 15842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:28.438567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.3:37383)
0928 08:47:28.439243 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:28.439319 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":465,"threads_started":1}
I20260928 08:47:28.488797 15843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:28.488002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.1:38961)
0928 08:47:28.488524 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:28.488612 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":326,"threads_started":1}
I20260928 08:47:28.965677 15844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:28.964991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.2:60391)
0928 08:47:28.965407 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:28.965495 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":202,"threads_started":1}
I20260928 08:47:29.441329 15845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:29.440527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.3:58377)
0928 08:47:29.441054 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:29.441148 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":264,"threads_started":1}
I20260928 08:47:29.490567 15846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:29.489770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.1:39033)
0928 08:47:29.490276 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:29.490388 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":277,"threads_started":1}
I20260928 08:47:29.967484 15847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:29.966714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.2:59187)
0928 08:47:29.967201 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:29.967321 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":248,"threads_started":1}
I20260928 08:47:30.443068 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:30.442352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.3:50623)
0928 08:47:30.442825 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:30.442908 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":233,"threads_started":1}
I20260928 08:47:30.492558 15849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:30.491578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.1:52025)
0928 08:47:30.492096 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:30.492180 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":289,"threads_started":1}
I20260928 08:47:30.969285 15850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:30.968481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.2:53761)
0928 08:47:30.969001 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:30.969088 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":260,"threads_started":1}
I20260928 08:47:31.445106 15851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:31.444173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.3:36783)
0928 08:47:31.444770 (+ 597us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:31.444854 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":336,"threads_started":1}
I20260928 08:47:31.494112 15852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:31.493415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.1:42629)
0928 08:47:31.493854 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:31.493933 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":214,"threads_started":1}
I20260928 08:47:31.970866 15853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:31.970153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.2:40963)
0928 08:47:31.970576 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:31.970658 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":209,"threads_started":1}
I20260928 08:47:32.446693 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:32.445974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.3:60305)
0928 08:47:32.446443 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:32.446517 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":267,"threads_started":1}
I20260928 08:47:32.496243 15855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:32.495400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.1:47655)
0928 08:47:32.495916 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:32.496010 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":229,"threads_started":1}
I20260928 08:47:32.972769 15856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:32.972058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.2:48229)
0928 08:47:32.972492 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:32.972579 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":224,"threads_started":1}
I20260928 08:47:33.429488 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 15384
I20260928 08:47:33.448566 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:33.447791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:41033 (local address 127.0.97.3:44855)
0928 08:47:33.448297 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:33.448384 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":226,"threads_started":1}
I20260928 08:47:33.454746 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:33.430695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50628 (local address 127.0.97.1:35491)
0928 08:47:33.431620 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:33.431662 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:33.431712 (+ 50us) server_negotiation.cc:407] Connection header received
0928 08:47:33.433783 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:33.433793 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:33.434248 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:33.434623 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:33.436356 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:33.437788 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:33.439161 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:33.440001 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:33.440211 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:33.447524 (+ 7313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:33.447537 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:33.447561 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:33.447614 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:33.451145 (+ 3531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:33.451926 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:33.451935 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:33.451951 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:33.452128 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:33.452540 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:33.452545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:33.452554 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:33.453130 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:33.453461 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:33.453965 (+ 504us) server_negotiation.cc:299] Negotiation successful
0928 08:47:33.454419 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725,"thread_start_us":610,"threads_started":1}
I20260928 08:47:33.456307 15542 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:47:33.548619 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 15384
I20260928 08:47:33.566908 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 15550
I20260928 08:47:33.585117 15861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:33.567912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53226 (local address 127.0.97.2:41805)
0928 08:47:33.568488 (+ 576us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:33.568502 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:33.568546 (+ 44us) server_negotiation.cc:407] Connection header received
0928 08:47:33.570058 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:33.570066 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:33.570481 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:33.570832 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:33.572342 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:33.573779 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:33.575118 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:33.575879 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:33.576069 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:33.578587 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:33.578596 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:33.578613 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:33.578652 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:33.581934 (+ 3282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:33.582653 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:33.582660 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:33.582666 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:33.582829 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:33.583176 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:33.583180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:33.583182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:33.583700 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:33.584015 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:33.584409 (+ 394us) server_negotiation.cc:299] Negotiation successful
0928 08:47:33.584829 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":308,"threads_started":1}
I20260928 08:47:33.586668 15673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:47:33.671555 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 15550
I20260928 08:47:33.690625 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 15681
I20260928 08:47:33.709062 15863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:33.691684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57004 (local address 127.0.97.3:43385)
0928 08:47:33.692313 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:33.692326 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:33.692366 (+ 40us) server_negotiation.cc:407] Connection header received
0928 08:47:33.694008 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:33.694016 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:33.694443 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:33.694812 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:33.696337 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:33.697563 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:33.698866 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:33.699540 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:33.699751 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:33.702462 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:33.702470 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:33.702486 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:33.702523 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:33.705475 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:33.706215 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:33.706222 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:33.706229 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:33.706476 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:33.706896 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:33.706902 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:33.706905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:33.707433 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:33.707766 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:33.708202 (+ 436us) server_negotiation.cc:299] Negotiation successful
0928 08:47:33.708621 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":328,"threads_started":1}
I20260928 08:47:33.711038 15804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:47:33.798754 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 15681
2026-09-28T08:47:33Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24608 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:47:33 dist-test-slave-n6xs krb5kdc[15867](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:47:33 dist-test-slave-n6xs krb5kdc[15867](info): set up 2 sockets
Sep 28 08:47:33 dist-test-slave-n6xs krb5kdc[15867](info): commencing operation
krb5kdc: starting...
W20260928 08:47:34.900802 388 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:47:35 dist-test-slave-n6xs krb5kdc[15867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:47:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:47:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:47:35.139243 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46675
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:47:35.481024 15880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:47:35.481308 15880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:47:35.481397 15880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:47:35.488399 15880 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 08:47:35.488493 15880 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 08:47:35.488530 15880 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 08:47:35.488575 15880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:47:35.488629 15880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:47:35.488696 15880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:47:35.488742 15880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:47:35.488806 15880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:47:35.498659 15880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:47:35.499982 15880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:47:35.501329 15880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:47:35.513538 15885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:35.513487 15888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:35.513579 15886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:47:35.514468 15880 server_base.cc:1034] running on GCE node
I20260928 08:47:35.515530 15880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:47:35.517154 15880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:47:35.518440 15880 hybrid_clock.cc:648] HybridClock initialized: now 1790585255518368 us; error 64 us; skew 500 ppm
I20260928 08:47:35.518844 15880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:47:35 dist-test-slave-n6xs krb5kdc[15867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:47:35.522296 15880 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:47:35.523973 15880 webserver.cc:460] Webserver started at http://127.0.97.62:34085/ using document root <none> and password file <none>
I20260928 08:47:35.524726 15880 fs_manager.cc:359] Metadata directory not provided
I20260928 08:47:35.524865 15880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:47:35.525209 15880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:47:35.528563 15880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "d8c5003d43984eefb3f76ea34e47753e"
format_stamp: "Formatted at 2026-09-28 08:47:35 on dist-test-slave-n6xs"
server_key: "be3ca84b3e8b3f4c29b0c28ec60c08e5"
server_key_iv: "b415d4a977937e6ad249e15fe290953b"
server_key_version: "encryptionkey@0"
I20260928 08:47:35.529440 15880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "d8c5003d43984eefb3f76ea34e47753e"
format_stamp: "Formatted at 2026-09-28 08:47:35 on dist-test-slave-n6xs"
server_key: "be3ca84b3e8b3f4c29b0c28ec60c08e5"
server_key_iv: "b415d4a977937e6ad249e15fe290953b"
server_key_version: "encryptionkey@0"
I20260928 08:47:35.534986 15880 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:47:35.538818 15895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:35.540418 15880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:47:35.540598 15880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "d8c5003d43984eefb3f76ea34e47753e"
format_stamp: "Formatted at 2026-09-28 08:47:35 on dist-test-slave-n6xs"
server_key: "be3ca84b3e8b3f4c29b0c28ec60c08e5"
server_key_iv: "b415d4a977937e6ad249e15fe290953b"
server_key_version: "encryptionkey@0"
I20260928 08:47:35.540831 15880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:47:35.558270 15880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:47:35.562657 15880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:47:35.562942 15880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:47:35.581372 15880 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46675
I20260928 08:47:35.581390 15946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46675 every 8 connection(s)
I20260928 08:47:35.582783 15880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:47:35.586601 15947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:47:35.591434 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 15880
I20260928 08:47:35.591675 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:47:35.592442 388 external_mini_cluster.cc:1428] Setting key 9416826114a11566039ae8a4ec2622cf
I20260928 08:47:35.597803 15947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e: Bootstrap starting.
I20260928 08:47:35.601759 15947 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e: Neither blocks nor log segments found. Creating new log.
I20260928 08:47:35.603129 15947 log.cc:824] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e: Log is configured to *not* fsync() on all Append() calls
I20260928 08:47:35.606688 15947 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e: No bootstrap required, opened a new log
Sep 28 08:47:35 dist-test-slave-n6xs krb5kdc[15867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:47:35.615002 15947 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c5003d43984eefb3f76ea34e47753e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46675 } }
I20260928 08:47:35.615403 15947 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:47:35.615513 15947 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8c5003d43984eefb3f76ea34e47753e, State: Initialized, Role: FOLLOWER
I20260928 08:47:35.616207 15947 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c5003d43984eefb3f76ea34e47753e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46675 } }
I20260928 08:47:35.616449 15947 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:47:35.616606 15947 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:47:35.616806 15947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:47:35.618830 15950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:35.594654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44216 (local address 127.0.97.62:46675)
0928 08:47:35.595660 (+ 1006us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:35.595673 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:35.595735 (+ 62us) server_negotiation.cc:407] Connection header received
0928 08:47:35.597684 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:35.597709 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:35.598189 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:35.598592 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:35.600691 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:35.602179 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:35.604130 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:35.605070 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:35.605224 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:35.610274 (+ 5050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:35.610283 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:35.610308 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:35.610351 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:35.613792 (+ 3441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:35.614568 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:35.614579 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:35.614586 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:35.614746 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:35.615189 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:35.615193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:35.615195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:35.615982 (+ 787us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:35.616308 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:35.616895 (+ 587us) server_negotiation.cc:299] Negotiation successful
0928 08:47:35.617389 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":769,"thread_start_us":486,"threads_started":1}
I20260928 08:47:35.620167 15947 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c5003d43984eefb3f76ea34e47753e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46675 } }
I20260928 08:47:35.620721 15947 leader_election.cc:302] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8c5003d43984eefb3f76ea34e47753e; no voters:
I20260928 08:47:35.621356 15947 leader_election.cc:288] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:47:35.621551 15952 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:47:35.622720 15952 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [term 1 LEADER]: Becoming Leader. State: Replica: d8c5003d43984eefb3f76ea34e47753e, State: Running, Role: LEADER
I20260928 08:47:35.623207 15952 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c5003d43984eefb3f76ea34e47753e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46675 } }
I20260928 08:47:35.623484 15947 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:47:35.626366 15953 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8c5003d43984eefb3f76ea34e47753e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c5003d43984eefb3f76ea34e47753e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46675 } } }
I20260928 08:47:35.626837 15953 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [sys.catalog]: This master's current role is: LEADER
I20260928 08:47:35.626366 15954 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8c5003d43984eefb3f76ea34e47753e. Latest consensus state: current_term: 1 leader_uuid: "d8c5003d43984eefb3f76ea34e47753e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c5003d43984eefb3f76ea34e47753e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46675 } } }
I20260928 08:47:35.627195 15954 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [sys.catalog]: This master's current role is: LEADER
I20260928 08:47:35.628335 15959 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:47:35.632742 15959 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:47:35.641853 15959 catalog_manager.cc:1347] Generated new cluster ID: 1fe21506ceb94a12898c2c5747916494
I20260928 08:47:35.641999 15959 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:47:35.668207 15959 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:47:35.669660 15959 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:47:35.682783 15959 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e: Generated new TSK 0
I20260928 08:47:35.683651 15959 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:47:35.741689 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46675
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:47:36.081449 15973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:47:36.081739 15973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:47:36.081888 15973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:47:36.089279 15973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:47:36.089394 15973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:47:36.089555 15973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:47:36.099280 15973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:47:36.100600 15973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:47:36.101878 15973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:47:36.114997 15978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:36.114959 15981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:36.114921 15979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:47:36.115784 15973 server_base.cc:1034] running on GCE node
I20260928 08:47:36.116396 15973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:47:36.117336 15973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:47:36.118552 15973 hybrid_clock.cc:648] HybridClock initialized: now 1790585256118487 us; error 55 us; skew 500 ppm
I20260928 08:47:36.118865 15973 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:47:36 dist-test-slave-n6xs krb5kdc[15867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:47:36.122021 15973 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:47:36.123592 15973 webserver.cc:460] Webserver started at http://127.0.97.1:36359/ using document root <none> and password file <none>
I20260928 08:47:36.124341 15973 fs_manager.cc:359] Metadata directory not provided
I20260928 08:47:36.124482 15973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:47:36.124842 15973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:47:36.127682 15973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/data/instance:
uuid: "16395cd18c07415489f77aa172a80133"
format_stamp: "Formatted at 2026-09-28 08:47:36 on dist-test-slave-n6xs"
server_key: "29ff7f520798481cab12b239ba5f69d0"
server_key_iv: "ea648a53b3fcd812f1fe8b5425aab000"
server_key_version: "encryptionkey@0"
I20260928 08:47:36.128513 15973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/wal/instance:
uuid: "16395cd18c07415489f77aa172a80133"
format_stamp: "Formatted at 2026-09-28 08:47:36 on dist-test-slave-n6xs"
server_key: "29ff7f520798481cab12b239ba5f69d0"
server_key_iv: "ea648a53b3fcd812f1fe8b5425aab000"
server_key_version: "encryptionkey@0"
I20260928 08:47:36.133538 15973 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 08:47:36.137090 15988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:36.138401 15973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:47:36.138533 15973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/wal
uuid: "16395cd18c07415489f77aa172a80133"
format_stamp: "Formatted at 2026-09-28 08:47:36 on dist-test-slave-n6xs"
server_key: "29ff7f520798481cab12b239ba5f69d0"
server_key_iv: "ea648a53b3fcd812f1fe8b5425aab000"
server_key_version: "encryptionkey@0"
I20260928 08:47:36.138701 15973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:47:36.158373 15973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:47:36.162292 15973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:47:36.162520 15973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:47:36.163408 15973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:47:36.165112 15973 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:47:36.165182 15973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:36.165320 15973 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:47:36.165396 15973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:36.197980 15973 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42723
I20260928 08:47:36.198036 16100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42723 every 8 connection(s)
I20260928 08:47:36.199239 15973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:47:36.206027 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 15973
I20260928 08:47:36.206238 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:47:36.206888 388 external_mini_cluster.cc:1428] Setting key 03d555782db2623681389813907543fa
Sep 28 08:47:36 dist-test-slave-n6xs krb5kdc[15867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:47:36.220906 16103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:36.201834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:33641 (local address 127.0.97.62:46675)
0928 08:47:36.202277 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:36.202283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:36.203349 (+ 1066us) server_negotiation.cc:407] Connection header received
0928 08:47:36.205414 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:36.205418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:36.205486 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:36.205622 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:36.207309 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:36.208285 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:36.209862 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:36.210268 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:36.210402 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:36.213872 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:36.213877 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:36.213880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:36.213912 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:36.217373 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:36.218191 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:36.218199 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:36.218206 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:36.218360 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:36.218817 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:36.218825 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:36.218830 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:36.219198 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:36.219358 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:36.220134 (+ 776us) server_negotiation.cc:299] Negotiation successful
0928 08:47:36.220404 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":203,"threads_started":1}
I20260928 08:47:36.221580 16104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:36.202386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.1:33641)
0928 08:47:36.203168 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:36.203220 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:36.205131 (+ 1911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:36.205805 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:36.205813 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:36.206495 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:47:36.207128 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:36.207148 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:36.208453 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:36.208463 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:36.209715 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:36.209727 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:36.210514 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:36.210521 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:36.210751 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:36.211430 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:47:36.211457 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:47:36.213642 (+ 2185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:47:36.217532 (+ 3890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:36.217539 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:36.217553 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:36.218066 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:36.218503 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:36.218507 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:36.218511 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:36.218704 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:36.219326 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:47:36.219332 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:47:36.219675 (+ 343us) client_negotiation.cc:769] Sending connection context
0928 08:47:36.219982 (+ 307us) client_negotiation.cc:240] Negotiation successful
0928 08:47:36.220422 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":316,"threads_started":1}
I20260928 08:47:36.223137 16101 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46675
I20260928 08:47:36.223460 16101 heartbeater.cc:464] Registering TS with master...
I20260928 08:47:36.224344 16101 heartbeater.cc:511] Master 127.0.97.62:46675 requested a full tablet report, sending...
I20260928 08:47:36.226626 15912 ts_manager.cc:194] Registered new tserver with Master: 16395cd18c07415489f77aa172a80133 (127.0.97.1:42723)
I20260928 08:47:36.228266 15912 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:33641
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 08:47:36.263320 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46675
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:47:36.602030 16107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:47:36.602365 16107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:47:36.602542 16107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:47:36.610013 16107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:47:36.610186 16107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:47:36.610378 16107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:47:36.620628 16107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:47:36.622012 16107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:47:36.623435 16107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:47:36.637580 16112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:36.637624 16113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:36.637843 16115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:47:36.638510 16107 server_base.cc:1034] running on GCE node
I20260928 08:47:36.639137 16107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:47:36.640132 16107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:47:36.641362 16107 hybrid_clock.cc:648] HybridClock initialized: now 1790585256641315 us; error 40 us; skew 500 ppm
I20260928 08:47:36.641767 16107 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:47:36 dist-test-slave-n6xs krb5kdc[15867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:47:36.645265 16107 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:47:36.646771 16107 webserver.cc:460] Webserver started at http://127.0.97.2:40311/ using document root <none> and password file <none>
I20260928 08:47:36.647496 16107 fs_manager.cc:359] Metadata directory not provided
I20260928 08:47:36.647655 16107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:47:36.648036 16107 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:47:36.651059 16107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/data/instance:
uuid: "c018f42c29c74413abcfd6054fcb9be2"
format_stamp: "Formatted at 2026-09-28 08:47:36 on dist-test-slave-n6xs"
server_key: "e7acf1de1fa2ce7210205ae55746084e"
server_key_iv: "f6d85eabd0984b921b082970a1b2bf03"
server_key_version: "encryptionkey@0"
I20260928 08:47:36.651916 16107 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/wal/instance:
uuid: "c018f42c29c74413abcfd6054fcb9be2"
format_stamp: "Formatted at 2026-09-28 08:47:36 on dist-test-slave-n6xs"
server_key: "e7acf1de1fa2ce7210205ae55746084e"
server_key_iv: "f6d85eabd0984b921b082970a1b2bf03"
server_key_version: "encryptionkey@0"
I20260928 08:47:36.657253 16107 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 08:47:36.660919 16122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:36.662441 16107 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 08:47:36.662627 16107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/wal
uuid: "c018f42c29c74413abcfd6054fcb9be2"
format_stamp: "Formatted at 2026-09-28 08:47:36 on dist-test-slave-n6xs"
server_key: "e7acf1de1fa2ce7210205ae55746084e"
server_key_iv: "f6d85eabd0984b921b082970a1b2bf03"
server_key_version: "encryptionkey@0"
I20260928 08:47:36.662897 16107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:47:36.696797 16107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:47:36.701486 16107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:47:36.701782 16107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:47:36.702780 16107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:47:36.704162 16107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:47:36.704262 16107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:36.704411 16107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:47:36.704483 16107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:36.738293 16107 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45079
I20260928 08:47:36.738379 16234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45079 every 8 connection(s)
I20260928 08:47:36.739578 16107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:47:36.746495 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 16107
I20260928 08:47:36.746699 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:47:36.747156 388 external_mini_cluster.cc:1428] Setting key cd86dbf43588e4583a0a70cf7d6c2264
Sep 28 08:47:36 dist-test-slave-n6xs krb5kdc[15867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:47:36.762657 16237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:36.742297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:46351 (local address 127.0.97.62:46675)
0928 08:47:36.742829 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:36.742833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:36.743693 (+ 860us) server_negotiation.cc:407] Connection header received
0928 08:47:36.745812 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:36.745819 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:36.745918 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:36.746052 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:36.747960 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:36.749024 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:36.750710 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:36.751192 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:36.751346 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:36.755121 (+ 3775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:36.755127 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:36.755131 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:36.755166 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:36.758403 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:36.759307 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:36.759316 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:36.759322 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:36.759510 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:36.760158 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:36.760167 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:36.760173 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:36.760624 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:36.760794 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:36.761805 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0928 08:47:36.762099 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":308,"threads_started":1}
I20260928 08:47:36.763403 16238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:36.742766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.2:46351)
0928 08:47:36.743513 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:36.743562 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:36.745497 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:36.746302 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:36.746310 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:36.746989 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:47:36.747687 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:36.747715 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:36.749201 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:36.749207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:36.750546 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:36.750558 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:36.751443 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:36.751452 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:36.751706 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:36.752504 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:47:36.752535 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:47:36.754857 (+ 2322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:47:36.758599 (+ 3742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:36.758608 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:36.758624 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:36.759148 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:36.759664 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:36.759668 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:36.759671 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:36.759974 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:36.760797 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:47:36.760804 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:47:36.761293 (+ 489us) client_negotiation.cc:769] Sending connection context
0928 08:47:36.761647 (+ 354us) client_negotiation.cc:240] Negotiation successful
0928 08:47:36.762238 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":331,"threads_started":1}
I20260928 08:47:36.764997 16235 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46675
I20260928 08:47:36.765311 16235 heartbeater.cc:464] Registering TS with master...
I20260928 08:47:36.766067 16235 heartbeater.cc:511] Master 127.0.97.62:46675 requested a full tablet report, sending...
I20260928 08:47:36.767688 15912 ts_manager.cc:194] Registered new tserver with Master: c018f42c29c74413abcfd6054fcb9be2 (127.0.97.2:45079)
I20260928 08:47:36.768605 15912 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:46351
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:47:36.803488 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46675
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:47:37.130388 16241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:47:37.130681 16241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:47:37.130839 16241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:47:37.138572 16241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:47:37.138690 16241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:47:37.138909 16241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:47:37.149513 16241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:46675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:47:37.150877 16241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:47:37.152245 16241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:47:37.166255 16247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:37.166324 16246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:37.166524 16249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:47:37.166956 16241 server_base.cc:1034] running on GCE node
I20260928 08:47:37.167754 16241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:47:37.169114 16241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:47:37.170339 16241 hybrid_clock.cc:648] HybridClock initialized: now 1790585257170298 us; error 63 us; skew 500 ppm
I20260928 08:47:37.170686 16241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:47:37 dist-test-slave-n6xs krb5kdc[15867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:47:37.174005 16241 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:47:37.175493 16241 webserver.cc:460] Webserver started at http://127.0.97.3:37671/ using document root <none> and password file <none>
I20260928 08:47:37.176231 16241 fs_manager.cc:359] Metadata directory not provided
I20260928 08:47:37.176337 16241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:47:37.176589 16241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:47:37.179431 16241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/data/instance:
uuid: "59dc27f1dc6847b5acee11d95e81ba0f"
format_stamp: "Formatted at 2026-09-28 08:47:37 on dist-test-slave-n6xs"
server_key: "b45b76e3326428c205094c335aee1f6a"
server_key_iv: "b886b98f904466ac8534631e81540f77"
server_key_version: "encryptionkey@0"
I20260928 08:47:37.180208 16241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/wal/instance:
uuid: "59dc27f1dc6847b5acee11d95e81ba0f"
format_stamp: "Formatted at 2026-09-28 08:47:37 on dist-test-slave-n6xs"
server_key: "b45b76e3326428c205094c335aee1f6a"
server_key_iv: "b886b98f904466ac8534631e81540f77"
server_key_version: "encryptionkey@0"
I20260928 08:47:37.185374 16241 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 08:47:37.188771 16256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:37.190164 16241 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260928 08:47:37.190340 16241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/wal
uuid: "59dc27f1dc6847b5acee11d95e81ba0f"
format_stamp: "Formatted at 2026-09-28 08:47:37 on dist-test-slave-n6xs"
server_key: "b45b76e3326428c205094c335aee1f6a"
server_key_iv: "b886b98f904466ac8534631e81540f77"
server_key_version: "encryptionkey@0"
I20260928 08:47:37.190548 16241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:47:37.210932 16241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:47:37.215305 16241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:47:37.215595 16241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:47:37.216743 16241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:47:37.218034 16241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:47:37.218140 16241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:37.218286 16241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:47:37.218358 16241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:37.232100 16101 heartbeater.cc:503] Master 127.0.97.62:46675 was elected leader, sending a full tablet report...
I20260928 08:47:37.254626 16241 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42469
I20260928 08:47:37.254712 16368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42469 every 8 connection(s)
I20260928 08:47:37.256067 16241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:47:37.265617 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 16241
I20260928 08:47:37.265825 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:47:37.266338 388 external_mini_cluster.cc:1428] Setting key 9e715cc9184e02e82f23661970c43540
Sep 28 08:47:37 dist-test-slave-n6xs krb5kdc[15867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:47:37.277046 16237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:37.258655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35657 (local address 127.0.97.62:46675)
0928 08:47:37.258874 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:37.258881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:37.260076 (+ 1195us) server_negotiation.cc:407] Connection header received
0928 08:47:37.261875 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:37.261879 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:37.261951 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:37.262056 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:37.263610 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:37.264474 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:37.266252 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:37.266804 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:37.266946 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:37.271346 (+ 4400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:37.271351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:37.271354 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:37.271419 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:37.274170 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:37.274922 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:37.274926 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:37.274928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:37.275082 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:37.275516 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:37.275519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:37.275521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:37.275789 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:37.275963 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:37.276628 (+ 665us) server_negotiation.cc:299] Negotiation successful
0928 08:47:37.276808 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 08:47:37.278067 16371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:37.259124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.3:35657)
0928 08:47:37.259857 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:37.259898 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:37.261585 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:37.262265 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:37.262274 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:37.262826 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:47:37.263423 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:37.263440 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:37.264634 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:37.264638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:37.266077 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:37.266089 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:37.267061 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:37.267069 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:37.267294 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:37.268212 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:47:37.268239 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:47:37.271123 (+ 2884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:47:37.274312 (+ 3189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:37.274320 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:37.274333 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:37.274811 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:37.275213 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:37.275218 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:37.275220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:37.275410 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:37.275959 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:47:37.275965 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:47:37.276286 (+ 321us) client_negotiation.cc:769] Sending connection context
0928 08:47:37.276535 (+ 249us) client_negotiation.cc:240] Negotiation successful
0928 08:47:37.276937 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":278,"threads_started":1}
I20260928 08:47:37.279606 16369 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46675
I20260928 08:47:37.280035 16369 heartbeater.cc:464] Registering TS with master...
I20260928 08:47:37.280985 16369 heartbeater.cc:511] Master 127.0.97.62:46675 requested a full tablet report, sending...
I20260928 08:47:37.282716 15912 ts_manager.cc:194] Registered new tserver with Master: 59dc27f1dc6847b5acee11d95e81ba0f (127.0.97.3:42469)
I20260928 08:47:37.283545 15912 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:35657
I20260928 08:47:37.295125 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:47:37.312800 16237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:37.299766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44230 (local address 127.0.97.62:46675)
0928 08:47:37.300006 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:37.300016 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:37.300391 (+ 375us) server_negotiation.cc:407] Connection header received
0928 08:47:37.300533 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:37.300536 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:37.300600 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:37.300686 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:37.302552 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:37.303323 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:37.304839 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:37.305237 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:37.305331 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:37.307651 (+ 2320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:37.307657 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:37.307660 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:37.307692 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:37.310219 (+ 2527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:37.310852 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:37.310856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:37.310858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:37.310972 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:37.311374 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:37.311381 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:37.311387 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:37.311788 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:37.312038 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:37.312363 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 08:47:37.312559 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 08:47:37.317023 15912 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44230:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:47:37.324239 15912 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:47:37.361544 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:37.337046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49932 (local address 127.0.97.2:45079)
0928 08:47:37.352129 (+ 15083us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:37.352141 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:37.352176 (+ 35us) server_negotiation.cc:407] Connection header received
0928 08:47:37.352291 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:37.352299 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:37.352576 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:37.352818 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:47:37.354438 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:37.356079 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:37.358221 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:37.360412 (+ 2191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:37.360561 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:37.360765 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:37.361043 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 08:47:37.361205 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14878,"thread_start_us":14760,"threads_started":1}
I20260928 08:47:37.361651 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:37.336811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45079 (local address 127.0.0.1:49932)
0928 08:47:37.350828 (+ 14017us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:37.350901 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:37.351118 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:37.353704 (+ 2586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:37.353711 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:37.353741 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:47:37.354129 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:37.354140 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:37.356245 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:37.356251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:37.358058 (+ 1807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:37.358070 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:37.360594 (+ 2524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:37.360600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:37.360777 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:37.360839 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 08:47:37.360985 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 08:47:37.361197 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13680,"mutex_wait_us":31,"thread_start_us":14618,"threads_started":1}
I20260928 08:47:37.362311 16381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:37.337022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42723 (local address 127.0.0.1:54804)
0928 08:47:37.351729 (+ 14707us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:37.351750 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:37.351915 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:37.354093 (+ 2178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:37.354097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:37.354118 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:47:37.354753 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:37.354765 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:37.356595 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:37.356605 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:37.358833 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:37.358847 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:37.361064 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:37.361071 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:37.361590 (+ 519us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:37.361623 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 08:47:37.361711 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 08:47:37.361846 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14551,"mutex_wait_us":346,"thread_start_us":13317,"threads_started":1}
I20260928 08:47:37.362488 16385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:37.340091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42469 (local address 127.0.0.1:50272)
0928 08:47:37.352952 (+ 12861us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:37.352970 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:37.353087 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:37.354056 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:37.354064 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:37.354096 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:47:37.354710 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:37.354721 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:37.356152 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:37.356158 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:37.358768 (+ 2610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:37.358784 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:37.361032 (+ 2248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:37.361044 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:37.361292 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:37.361324 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 08:47:37.361415 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 08:47:37.361560 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12748,"mutex_wait_us":220,"thread_start_us":12568,"threads_started":1}
I20260928 08:47:37.363197 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:37.337706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54804 (local address 127.0.97.1:42723)
0928 08:47:37.353519 (+ 15813us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:37.353527 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:37.353553 (+ 26us) server_negotiation.cc:407] Connection header received
0928 08:47:37.353651 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:37.353661 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:37.353863 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:37.354210 (+ 347us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:47:37.354963 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:37.356461 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:37.358992 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:37.360820 (+ 1828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:37.361312 (+ 492us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:37.361532 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:37.362705 (+ 1173us) server_negotiation.cc:299] Negotiation successful
0928 08:47:37.362870 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15656,"thread_start_us":15481,"threads_started":1}
I20260928 08:47:37.363396 16386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:37.340590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50272 (local address 127.0.97.3:42469)
0928 08:47:37.352518 (+ 11928us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:37.352527 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:37.353215 (+ 688us) server_negotiation.cc:407] Connection header received
0928 08:47:37.353334 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:37.353341 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:37.353577 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:37.353757 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:47:37.354886 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:37.356021 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:37.358963 (+ 2942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:37.360518 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:37.360643 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:37.360869 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:37.362875 (+ 2006us) server_negotiation.cc:299] Negotiation successful
0928 08:47:37.363036 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11750,"thread_start_us":11564,"threads_started":1}
I20260928 08:47:37.365458 16170 tablet_service.cc:1467] Processing CreateTablet for tablet a8759cd8eb6146b08106d278e997ed21 (DEFAULT_TABLE table=test-table [id=5e45335cf8d140ac958a453dabd390e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:47:37.366977 16170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8759cd8eb6146b08106d278e997ed21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:47:37.366737 16036 tablet_service.cc:1467] Processing CreateTablet for tablet a8759cd8eb6146b08106d278e997ed21 (DEFAULT_TABLE table=test-table [id=5e45335cf8d140ac958a453dabd390e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:47:37.366729 16304 tablet_service.cc:1467] Processing CreateTablet for tablet a8759cd8eb6146b08106d278e997ed21 (DEFAULT_TABLE table=test-table [id=5e45335cf8d140ac958a453dabd390e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:47:37.368203 16304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8759cd8eb6146b08106d278e997ed21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:47:37.368209 16036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8759cd8eb6146b08106d278e997ed21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:47:37.378269 16387 tablet_bootstrap.cc:492] T a8759cd8eb6146b08106d278e997ed21 P c018f42c29c74413abcfd6054fcb9be2: Bootstrap starting.
I20260928 08:47:37.381639 16387 tablet_bootstrap.cc:654] T a8759cd8eb6146b08106d278e997ed21 P c018f42c29c74413abcfd6054fcb9be2: Neither blocks nor log segments found. Creating new log.
I20260928 08:47:37.382548 16388 tablet_bootstrap.cc:492] T a8759cd8eb6146b08106d278e997ed21 P 16395cd18c07415489f77aa172a80133: Bootstrap starting.
I20260928 08:47:37.383046 16387 log.cc:824] T a8759cd8eb6146b08106d278e997ed21 P c018f42c29c74413abcfd6054fcb9be2: Log is configured to *not* fsync() on all Append() calls
I20260928 08:47:37.383435 16389 tablet_bootstrap.cc:492] T a8759cd8eb6146b08106d278e997ed21 P 59dc27f1dc6847b5acee11d95e81ba0f: Bootstrap starting.
I20260928 08:47:37.387014 16388 tablet_bootstrap.cc:654] T a8759cd8eb6146b08106d278e997ed21 P 16395cd18c07415489f77aa172a80133: Neither blocks nor log segments found. Creating new log.
I20260928 08:47:37.387673 16389 tablet_bootstrap.cc:654] T a8759cd8eb6146b08106d278e997ed21 P 59dc27f1dc6847b5acee11d95e81ba0f: Neither blocks nor log segments found. Creating new log.
I20260928 08:47:37.388136 16388 log.cc:824] T a8759cd8eb6146b08106d278e997ed21 P 16395cd18c07415489f77aa172a80133: Log is configured to *not* fsync() on all Append() calls
I20260928 08:47:37.389221 16389 log.cc:824] T a8759cd8eb6146b08106d278e997ed21 P 59dc27f1dc6847b5acee11d95e81ba0f: Log is configured to *not* fsync() on all Append() calls
I20260928 08:47:37.391183 16388 tablet_bootstrap.cc:492] T a8759cd8eb6146b08106d278e997ed21 P 16395cd18c07415489f77aa172a80133: No bootstrap required, opened a new log
I20260928 08:47:37.391467 16388 ts_tablet_manager.cc:1397] T a8759cd8eb6146b08106d278e997ed21 P 16395cd18c07415489f77aa172a80133: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260928 08:47:37.391669 16387 tablet_bootstrap.cc:492] T a8759cd8eb6146b08106d278e997ed21 P c018f42c29c74413abcfd6054fcb9be2: No bootstrap required, opened a new log
I20260928 08:47:37.391978 16387 ts_tablet_manager.cc:1397] T a8759cd8eb6146b08106d278e997ed21 P c018f42c29c74413abcfd6054fcb9be2: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.007s
I20260928 08:47:37.392505 16389 tablet_bootstrap.cc:492] T a8759cd8eb6146b08106d278e997ed21 P 59dc27f1dc6847b5acee11d95e81ba0f: No bootstrap required, opened a new log
I20260928 08:47:37.392762 16389 ts_tablet_manager.cc:1397] T a8759cd8eb6146b08106d278e997ed21 P 59dc27f1dc6847b5acee11d95e81ba0f: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260928 08:47:37.399428 16387 raft_consensus.cc:348] T a8759cd8eb6146b08106d278e997ed21 P c018f42c29c74413abcfd6054fcb9be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c018f42c29c74413abcfd6054fcb9be2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45079 } } peers { permanent_uuid: "16395cd18c07415489f77aa172a80133" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42723 } } peers { permanent_uuid: "59dc27f1dc6847b5acee11d95e81ba0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42469 } }
I20260928 08:47:37.399942 16387 raft_consensus.cc:374] T a8759cd8eb6146b08106d278e997ed21 P c018f42c29c74413abcfd6054fcb9be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:47:37.400040 16387 raft_consensus.cc:729] T a8759cd8eb6146b08106d278e997ed21 P c018f42c29c74413abcfd6054fcb9be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c018f42c29c74413abcfd6054fcb9be2, State: Initialized, Role: FOLLOWER
I20260928 08:47:37.400671 16387 consensus_queue.cc:260] T a8759cd8eb6146b08106d278e997ed21 P c018f42c29c74413abcfd6054fcb9be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c018f42c29c74413abcfd6054fcb9be2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45079 } } peers { permanent_uuid: "16395cd18c07415489f77aa172a80133" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42723 } } peers { permanent_uuid: "59dc27f1dc6847b5acee11d95e81ba0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42469 } }
I20260928 08:47:37.401749 16235 heartbeater.cc:503] Master 127.0.97.62:46675 was elected leader, sending a full tablet report...
I20260928 08:47:37.402452 16387 ts_tablet_manager.cc:1428] T a8759cd8eb6146b08106d278e997ed21 P c018f42c29c74413abcfd6054fcb9be2: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20260928 08:47:37.402249 16388 raft_consensus.cc:348] T a8759cd8eb6146b08106d278e997ed21 P 16395cd18c07415489f77aa172a80133 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c018f42c29c74413abcfd6054fcb9be2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45079 } } peers { permanent_uuid: "16395cd18c07415489f77aa172a80133" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42723 } } peers { permanent_uuid: "59dc27f1dc6847b5acee11d95e81ba0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42469 } }
I20260928 08:47:37.402663 16388 raft_consensus.cc:374] T a8759cd8eb6146b08106d278e997ed21 P 16395cd18c07415489f77aa172a80133 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:47:37.402743 16388 raft_consensus.cc:729] T a8759cd8eb6146b08106d278e997ed21 P 16395cd18c07415489f77aa172a80133 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16395cd18c07415489f77aa172a80133, State: Initialized, Role: FOLLOWER
I20260928 08:47:37.403287 16388 consensus_queue.cc:260] T a8759cd8eb6146b08106d278e997ed21 P 16395cd18c07415489f77aa172a80133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c018f42c29c74413abcfd6054fcb9be2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45079 } } peers { permanent_uuid: "16395cd18c07415489f77aa172a80133" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42723 } } peers { permanent_uuid: "59dc27f1dc6847b5acee11d95e81ba0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42469 } }
I20260928 08:47:37.404276 16389 raft_consensus.cc:348] T a8759cd8eb6146b08106d278e997ed21 P 59dc27f1dc6847b5acee11d95e81ba0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c018f42c29c74413abcfd6054fcb9be2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45079 } } peers { permanent_uuid: "16395cd18c07415489f77aa172a80133" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42723 } } peers { permanent_uuid: "59dc27f1dc6847b5acee11d95e81ba0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42469 } }
I20260928 08:47:37.404673 16389 raft_consensus.cc:374] T a8759cd8eb6146b08106d278e997ed21 P 59dc27f1dc6847b5acee11d95e81ba0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:47:37.404768 16389 raft_consensus.cc:729] T a8759cd8eb6146b08106d278e997ed21 P 59dc27f1dc6847b5acee11d95e81ba0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59dc27f1dc6847b5acee11d95e81ba0f, State: Initialized, Role: FOLLOWER
I20260928 08:47:37.405081 16388 ts_tablet_manager.cc:1428] T a8759cd8eb6146b08106d278e997ed21 P 16395cd18c07415489f77aa172a80133: Time spent starting tablet: real 0.013s user 0.011s sys 0.003s
I20260928 08:47:37.405331 16389 consensus_queue.cc:260] T a8759cd8eb6146b08106d278e997ed21 P 59dc27f1dc6847b5acee11d95e81ba0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c018f42c29c74413abcfd6054fcb9be2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45079 } } peers { permanent_uuid: "16395cd18c07415489f77aa172a80133" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42723 } } peers { permanent_uuid: "59dc27f1dc6847b5acee11d95e81ba0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42469 } }
I20260928 08:47:37.406747 16369 heartbeater.cc:503] Master 127.0.97.62:46675 was elected leader, sending a full tablet report...
I20260928 08:47:37.408535 16389 ts_tablet_manager.cc:1428] T a8759cd8eb6146b08106d278e997ed21 P 59dc27f1dc6847b5acee11d95e81ba0f: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
W20260928 08:47:37.451591 16102 tablet.cc:2369] T a8759cd8eb6146b08106d278e997ed21 P 16395cd18c07415489f77aa172a80133: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:47:37.491745 16236 tablet.cc:2369] T a8759cd8eb6146b08106d278e997ed21 P c018f42c29c74413abcfd6054fcb9be2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:47:37.507706 16370 tablet.cc:2369] T a8759cd8eb6146b08106d278e997ed21 P 59dc27f1dc6847b5acee11d95e81ba0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:47:37.630386 16395 raft_consensus.cc:482] T a8759cd8eb6146b08106d278e997ed21 P 59dc27f1dc6847b5acee11d95e81ba0f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:47:37.630740 16395 raft_consensus.cc:504] T a8759cd8eb6146b08106d278e997ed21 P 59dc27f1dc6847b5acee11d95e81ba0f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c018f42c29c74413abcfd6054fcb9be2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45079 } } peers { permanent_uuid: "16395cd18c07415489f77aa172a80133" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42723 } } peers { permanent_uuid: "59dc27f1dc6847b5acee11d95e81ba0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42469 } }
I20260928 08:47:37.632577 16395 leader_election.cc:288] T a8759cd8eb6146b08106d278e997ed21 P 59dc27f1dc6847b5acee11d95e81ba0f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c018f42c29c74413abcfd6054fcb9be2 (127.0.97.2:45079), 16395cd18c07415489f77aa172a80133 (127.0.97.1:42723)
I20260928 08:47:37.638916 16371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:37.632889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42723 (local address 127.0.97.3:38347)
0928 08:47:37.633253 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:37.633274 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:37.633393 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:37.633791 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:37.633797 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:37.633824 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:47:37.634061 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:37.634075 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:37.635514 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:37.635517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:37.637023 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:37.637035 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:37.638155 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:37.638160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:37.638352 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:37.638382 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 08:47:37.638482 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 08:47:37.638614 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":45}
I20260928 08:47:37.638895 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:37.633163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:38347 (local address 127.0.97.1:42723)
0928 08:47:37.633416 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:37.633425 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:37.633450 (+ 25us) server_negotiation.cc:407] Connection header received
0928 08:47:37.633558 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:37.633566 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:37.633640 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:37.633744 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:47:37.634199 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:37.635348 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:37.637200 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:37.637995 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:37.638145 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:37.638279 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:37.638527 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 08:47:37.638650 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 08:47:37.639510 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:37.633114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:33677 (local address 127.0.97.2:45079)
0928 08:47:37.633298 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:37.633306 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:37.633981 (+ 675us) server_negotiation.cc:407] Connection header received
0928 08:47:37.634149 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:37.634154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:37.634232 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:37.634355 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:47:37.634943 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:37.636005 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:37.637722 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:37.638546 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:37.638714 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:37.638807 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:37.639137 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 08:47:37.639262 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 08:47:37.640244 16396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:37.632936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45079 (local address 127.0.97.3:33677)
0928 08:47:37.633866 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:37.633893 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:37.634010 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:37.634388 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:37.634391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:37.634409 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:47:37.634725 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:37.634733 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:37.636154 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:37.636160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:37.637557 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:37.637569 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:37.638758 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:37.638764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:37.638925 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:37.638947 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 08:47:37.639702 (+ 755us) client_negotiation.cc:240] Negotiation successful
0928 08:47:37.639859 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"mutex_wait_us":29,"thread_start_us":590,"threads_started":1}
I20260928 08:47:37.640045 16056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8759cd8eb6146b08106d278e997ed21" candidate_uuid: "59dc27f1dc6847b5acee11d95e81ba0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16395cd18c07415489f77aa172a80133" is_pre_election: true
I20260928 08:47:37.640652 16056 raft_consensus.cc:2454] T a8759cd8eb6146b08106d278e997ed21 P 16395cd18c07415489f77aa172a80133 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59dc27f1dc6847b5acee11d95e81ba0f in term 0.
I20260928 08:47:37.641229 16190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8759cd8eb6146b08106d278e997ed21" candidate_uuid: "59dc27f1dc6847b5acee11d95e81ba0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c018f42c29c74413abcfd6054fcb9be2" is_pre_election: true
I20260928 08:47:37.641585 16258 leader_election.cc:302] T a8759cd8eb6146b08106d278e997ed21 P 59dc27f1dc6847b5acee11d95e81ba0f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16395cd18c07415489f77aa172a80133, 59dc27f1dc6847b5acee11d95e81ba0f; no voters:
I20260928 08:47:37.641726 16190 raft_consensus.cc:2454] T a8759cd8eb6146b08106d278e997ed21 P c018f42c29c74413abcfd6054fcb9be2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59dc27f1dc6847b5acee11d95e81ba0f in term 0.
I20260928 08:47:37.642032 16395 raft_consensus.cc:2781] T a8759cd8eb6146b08106d278e997ed21 P 59dc27f1dc6847b5acee11d95e81ba0f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:47:37.642215 16395 raft_consensus.cc:482] T a8759cd8eb6146b08106d278e997ed21 P 59dc27f1dc6847b5acee11d95e81ba0f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:47:37.642351 16395 raft_consensus.cc:3037] T a8759cd8eb6146b08106d278e997ed21 P 59dc27f1dc6847b5acee11d95e81ba0f [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:47:37.644761 16395 raft_consensus.cc:504] T a8759cd8eb6146b08106d278e997ed21 P 59dc27f1dc6847b5acee11d95e81ba0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c018f42c29c74413abcfd6054fcb9be2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45079 } } peers { permanent_uuid: "16395cd18c07415489f77aa172a80133" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42723 } } peers { permanent_uuid: "59dc27f1dc6847b5acee11d95e81ba0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42469 } }
I20260928 08:47:37.645594 16395 leader_election.cc:288] T a8759cd8eb6146b08106d278e997ed21 P 59dc27f1dc6847b5acee11d95e81ba0f [CANDIDATE]: Term 1 election: Requested vote from peers c018f42c29c74413abcfd6054fcb9be2 (127.0.97.2:45079), 16395cd18c07415489f77aa172a80133 (127.0.97.1:42723)
I20260928 08:47:37.646003 16056 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8759cd8eb6146b08106d278e997ed21" candidate_uuid: "59dc27f1dc6847b5acee11d95e81ba0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16395cd18c07415489f77aa172a80133"
I20260928 08:47:37.646103 16190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8759cd8eb6146b08106d278e997ed21" candidate_uuid: "59dc27f1dc6847b5acee11d95e81ba0f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c018f42c29c74413abcfd6054fcb9be2"
I20260928 08:47:37.646203 16056 raft_consensus.cc:3037] T a8759cd8eb6146b08106d278e997ed21 P 16395cd18c07415489f77aa172a80133 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:47:37.646350 16190 raft_consensus.cc:3037] T a8759cd8eb6146b08106d278e997ed21 P c018f42c29c74413abcfd6054fcb9be2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:47:37.648648 16056 raft_consensus.cc:2454] T a8759cd8eb6146b08106d278e997ed21 P 16395cd18c07415489f77aa172a80133 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59dc27f1dc6847b5acee11d95e81ba0f in term 1.
I20260928 08:47:37.648759 16190 raft_consensus.cc:2454] T a8759cd8eb6146b08106d278e997ed21 P c018f42c29c74413abcfd6054fcb9be2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59dc27f1dc6847b5acee11d95e81ba0f in term 1.
I20260928 08:47:37.649134 16258 leader_election.cc:302] T a8759cd8eb6146b08106d278e997ed21 P 59dc27f1dc6847b5acee11d95e81ba0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16395cd18c07415489f77aa172a80133, 59dc27f1dc6847b5acee11d95e81ba0f; no voters:
I20260928 08:47:37.649462 16395 raft_consensus.cc:2781] T a8759cd8eb6146b08106d278e997ed21 P 59dc27f1dc6847b5acee11d95e81ba0f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:47:37.649935 16395 raft_consensus.cc:686] T a8759cd8eb6146b08106d278e997ed21 P 59dc27f1dc6847b5acee11d95e81ba0f [term 1 LEADER]: Becoming Leader. State: Replica: 59dc27f1dc6847b5acee11d95e81ba0f, State: Running, Role: LEADER
I20260928 08:47:37.650413 16395 consensus_queue.cc:237] T a8759cd8eb6146b08106d278e997ed21 P 59dc27f1dc6847b5acee11d95e81ba0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c018f42c29c74413abcfd6054fcb9be2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45079 } } peers { permanent_uuid: "16395cd18c07415489f77aa172a80133" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42723 } } peers { permanent_uuid: "59dc27f1dc6847b5acee11d95e81ba0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42469 } }
I20260928 08:47:37.655128 15910 catalog_manager.cc:5495] T a8759cd8eb6146b08106d278e997ed21 P 59dc27f1dc6847b5acee11d95e81ba0f reported cstate change: term changed from 0 to 1, leader changed from <none> to 59dc27f1dc6847b5acee11d95e81ba0f (127.0.97.3). New cstate: current_term: 1 leader_uuid: "59dc27f1dc6847b5acee11d95e81ba0f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c018f42c29c74413abcfd6054fcb9be2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16395cd18c07415489f77aa172a80133" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59dc27f1dc6847b5acee11d95e81ba0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42469 } health_report { overall_health: HEALTHY } } }
I20260928 08:47:37.714721 16386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:37.709002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50278 (local address 127.0.97.3:42469)
0928 08:47:37.709183 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:37.709188 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:37.709204 (+ 16us) server_negotiation.cc:407] Connection header received
0928 08:47:37.709291 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:37.709294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:37.709376 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:37.709509 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:47:37.709866 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:37.710570 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:37.712177 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:37.712803 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:37.712952 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:37.713490 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:47:37.714089 (+ 599us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:47:37.714213 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:37.714331 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 08:47:37.714405 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 08:47:37.723904 16190 raft_consensus.cc:1261] T a8759cd8eb6146b08106d278e997ed21 P c018f42c29c74413abcfd6054fcb9be2 [term 1 FOLLOWER]: Refusing update from remote peer 59dc27f1dc6847b5acee11d95e81ba0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:47:37.723904 16056 raft_consensus.cc:1261] T a8759cd8eb6146b08106d278e997ed21 P 16395cd18c07415489f77aa172a80133 [term 1 FOLLOWER]: Refusing update from remote peer 59dc27f1dc6847b5acee11d95e81ba0f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:47:37.725205 16397 consensus_queue.cc:1035] T a8759cd8eb6146b08106d278e997ed21 P 59dc27f1dc6847b5acee11d95e81ba0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c018f42c29c74413abcfd6054fcb9be2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:47:37.725579 16395 consensus_queue.cc:1035] T a8759cd8eb6146b08106d278e997ed21 P 59dc27f1dc6847b5acee11d95e81ba0f [LEADER]: Connected to new peer: Peer: permanent_uuid: "16395cd18c07415489f77aa172a80133" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:47:37.766986 15910 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:47:37.767302 15910 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:47:37.771770 15910 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e: Sending DeleteTablet for 3 replicas of tablet a8759cd8eb6146b08106d278e997ed21
I20260928 08:47:37.772965 16170 tablet_service.cc:1514] Processing DeleteTablet for tablet a8759cd8eb6146b08106d278e997ed21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:47:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49932
I20260928 08:47:37.774344 16036 tablet_service.cc:1514] Processing DeleteTablet for tablet a8759cd8eb6146b08106d278e997ed21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:47:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:54804
I20260928 08:47:37.774799 16410 tablet_replica.cc:330] T a8759cd8eb6146b08106d278e997ed21 P c018f42c29c74413abcfd6054fcb9be2: stopping tablet replica
I20260928 08:47:37.775224 16410 raft_consensus.cc:2229] T a8759cd8eb6146b08106d278e997ed21 P c018f42c29c74413abcfd6054fcb9be2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:47:37.775641 16410 raft_consensus.cc:2258] T a8759cd8eb6146b08106d278e997ed21 P c018f42c29c74413abcfd6054fcb9be2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:47:37.777081 16304 tablet_service.cc:1514] Processing DeleteTablet for tablet a8759cd8eb6146b08106d278e997ed21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:47:37 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:50272
I20260928 08:47:37.777427 16411 tablet_replica.cc:330] T a8759cd8eb6146b08106d278e997ed21 P 16395cd18c07415489f77aa172a80133: stopping tablet replica
I20260928 08:47:37.777868 16411 raft_consensus.cc:2229] T a8759cd8eb6146b08106d278e997ed21 P 16395cd18c07415489f77aa172a80133 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:47:37.778179 16412 tablet_replica.cc:330] T a8759cd8eb6146b08106d278e997ed21 P 59dc27f1dc6847b5acee11d95e81ba0f: stopping tablet replica
I20260928 08:47:37.778228 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 15973
I20260928 08:47:37.778358 16411 raft_consensus.cc:2258] T a8759cd8eb6146b08106d278e997ed21 P 16395cd18c07415489f77aa172a80133 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:47:37.778527 16412 raft_consensus.cc:2229] T a8759cd8eb6146b08106d278e997ed21 P 59dc27f1dc6847b5acee11d95e81ba0f [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:47:37.779135 16412 raft_consensus.cc:2258] T a8759cd8eb6146b08106d278e997ed21 P 59dc27f1dc6847b5acee11d95e81ba0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:47:37.780645 16410 ts_tablet_manager.cc:1905] T a8759cd8eb6146b08106d278e997ed21 P c018f42c29c74413abcfd6054fcb9be2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:47:37.785207 16412 ts_tablet_manager.cc:1905] T a8759cd8eb6146b08106d278e997ed21 P 59dc27f1dc6847b5acee11d95e81ba0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:47:37.786940 16411 ts_tablet_manager.cc:1905] T a8759cd8eb6146b08106d278e997ed21 P 16395cd18c07415489f77aa172a80133: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:47:37.788878 16410 ts_tablet_manager.cc:1918] T a8759cd8eb6146b08106d278e997ed21 P c018f42c29c74413abcfd6054fcb9be2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:47:37.789042 16410 log.cc:1196] T a8759cd8eb6146b08106d278e997ed21 P c018f42c29c74413abcfd6054fcb9be2: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/wal/wals/a8759cd8eb6146b08106d278e997ed21
I20260928 08:47:37.789641 16410 ts_tablet_manager.cc:1939] T a8759cd8eb6146b08106d278e997ed21 P c018f42c29c74413abcfd6054fcb9be2: Deleting consensus metadata
I20260928 08:47:37.791388 15899 catalog_manager.cc:4841] TS c018f42c29c74413abcfd6054fcb9be2 (127.0.97.2:45079): tablet a8759cd8eb6146b08106d278e997ed21 (table test-table [id=5e45335cf8d140ac958a453dabd390e8]) successfully deleted
I20260928 08:47:37.793004 16412 ts_tablet_manager.cc:1918] T a8759cd8eb6146b08106d278e997ed21 P 59dc27f1dc6847b5acee11d95e81ba0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:47:37.793239 16412 log.cc:1196] T a8759cd8eb6146b08106d278e997ed21 P 59dc27f1dc6847b5acee11d95e81ba0f: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/wal/wals/a8759cd8eb6146b08106d278e997ed21
I20260928 08:47:37.793823 16412 ts_tablet_manager.cc:1939] T a8759cd8eb6146b08106d278e997ed21 P 59dc27f1dc6847b5acee11d95e81ba0f: Deleting consensus metadata
I20260928 08:47:37.795239 16411 ts_tablet_manager.cc:1918] T a8759cd8eb6146b08106d278e997ed21 P 16395cd18c07415489f77aa172a80133: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:47:37.795408 16411 log.cc:1196] T a8759cd8eb6146b08106d278e997ed21 P 16395cd18c07415489f77aa172a80133: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/wal/wals/a8759cd8eb6146b08106d278e997ed21
I20260928 08:47:37.795575 15898 catalog_manager.cc:4841] TS 59dc27f1dc6847b5acee11d95e81ba0f (127.0.97.3:42469): tablet a8759cd8eb6146b08106d278e997ed21 (table test-table [id=5e45335cf8d140ac958a453dabd390e8]) successfully deleted
I20260928 08:47:37.795979 16411 ts_tablet_manager.cc:1939] T a8759cd8eb6146b08106d278e997ed21 P 16395cd18c07415489f77aa172a80133: Deleting consensus metadata
I20260928 08:47:37.797956 15897 catalog_manager.cc:4841] TS 16395cd18c07415489f77aa172a80133 (127.0.97.1:42723): tablet a8759cd8eb6146b08106d278e997ed21 (table test-table [id=5e45335cf8d140ac958a453dabd390e8]) successfully deleted
Sep 28 08:47:37 dist-test-slave-n6xs krb5kdc[15867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:47:37.812041 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:37.779624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54818 (local address 127.0.97.1:42723)
0928 08:47:37.780058 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:37.780068 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:37.795062 (+ 14994us) server_negotiation.cc:407] Connection header received
0928 08:47:37.795362 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:37.795369 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:37.795463 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:37.795554 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:37.797877 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:37.798726 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:37.800204 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:37.800619 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:37.800765 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:37.804967 (+ 4202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:37.804982 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:37.805002 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:37.805059 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:37.808826 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:37.809580 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:37.809590 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:37.809594 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:37.809760 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:37.810175 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:37.810181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:37.810187 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:37.810649 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:37.810955 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:37.811287 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 08:47:37.811525 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251}
I20260928 08:47:37.812793 16096 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:47:37.905028 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 15973
I20260928 08:47:37.923980 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 16107
Sep 28 08:47:37 dist-test-slave-n6xs krb5kdc[15867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 08:47:37.942978 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:37.925066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49934 (local address 127.0.97.2:45079)
0928 08:47:37.925277 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:37.925284 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:37.925306 (+ 22us) server_negotiation.cc:407] Connection header received
0928 08:47:37.925378 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:37.925383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:37.925465 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:37.925584 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:37.927309 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:37.928457 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:37.930054 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:37.930707 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:37.930840 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:37.935727 (+ 4887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:37.935747 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:37.935769 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:37.935851 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:37.939782 (+ 3931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:37.940651 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:37.940661 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:37.940665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:37.940840 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:37.941246 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:37.941251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:37.941255 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:37.941761 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:37.942074 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:37.942361 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 08:47:37.942600 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 08:47:37.943737 16230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:47:38.033823 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 16107
I20260928 08:47:38.052163 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 16241
Sep 28 08:47:38 dist-test-slave-n6xs krb5kdc[15867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 08:47:38.071957 16386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:38.052963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50284 (local address 127.0.97.3:42469)
0928 08:47:38.053140 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:38.053144 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:38.053228 (+ 84us) server_negotiation.cc:407] Connection header received
0928 08:47:38.053422 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:38.053427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:38.053494 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:38.053632 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:38.055645 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:38.056578 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:38.058459 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:38.058895 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:38.058992 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:38.064878 (+ 5886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:38.064892 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:38.064912 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:38.064964 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:38.068352 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:38.069359 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:38.069369 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:38.069371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:38.069593 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:38.070073 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:38.070078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:38.070080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:38.070482 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:38.070730 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:38.071188 (+ 458us) server_negotiation.cc:299] Negotiation successful
0928 08:47:38.071382 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 08:47:38.072678 16364 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:47:38.202158 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 16241
I20260928 08:47:38.228150 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 15880
I20260928 08:47:38.229626 15942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:47:38.344568 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 15880
I20260928 08:47:58.366074 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46675
--webserver_interface=127.0.97.62
--webserver_port=34085
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:47:58.705935 16418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:47:58.706250 16418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:47:58.706343 16418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:47:58.713631 16418 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 08:47:58.713747 16418 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 08:47:58.713788 16418 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 08:47:58.713832 16418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:47:58.713887 16418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:47:58.713960 16418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:47:58.714006 16418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:47:58.714058 16418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:47:58.724180 16418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:47:58.725533 16418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:47:58.726861 16418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:47:58.750911 16427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:58.750855 16425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:58.750890 16424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:47:58.753463 16418 server_base.cc:1034] running on GCE node
I20260928 08:47:58.754449 16418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:47:58.756242 16418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:47:58.757495 16418 hybrid_clock.cc:648] HybridClock initialized: now 1790585278757418 us; error 37 us; skew 500 ppm
I20260928 08:47:58.757977 16418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:47:58 dist-test-slave-n6xs krb5kdc[15867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585278, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:47:58.762059 16418 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:47:58.763794 16418 webserver.cc:460] Webserver started at http://127.0.97.62:34085/ using document root <none> and password file <none>
I20260928 08:47:58.764695 16418 fs_manager.cc:359] Metadata directory not provided
I20260928 08:47:58.764843 16418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:47:58.771015 16418 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.006s
I20260928 08:47:58.774199 16434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:58.775715 16418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:47:58.775947 16418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "d8c5003d43984eefb3f76ea34e47753e"
format_stamp: "Formatted at 2026-09-28 08:47:35 on dist-test-slave-n6xs"
server_key: "be3ca84b3e8b3f4c29b0c28ec60c08e5"
server_key_iv: "b415d4a977937e6ad249e15fe290953b"
server_key_version: "encryptionkey@0"
I20260928 08:47:58.777014 16418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:47:58.796046 16418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:47:58.800719 16418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:47:58.801040 16418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:47:58.821023 16418 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46675
I20260928 08:47:58.821071 16485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46675 every 8 connection(s)
I20260928 08:47:58.822783 16418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:47:58.830179 16486 sys_catalog.cc:263] Verifying existing consensus state
I20260928 08:47:58.832186 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 16418
I20260928 08:47:58.833006 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:42723
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46675
--webserver_port=36359
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:47:58.833835 16486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e: Bootstrap starting.
I20260928 08:47:58.854426 16486 log.cc:824] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e: Log is configured to *not* fsync() on all Append() calls
I20260928 08:47:58.866988 16486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:47:58.867599 16486 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e: Bootstrap complete.
I20260928 08:47:58.877238 16486 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c5003d43984eefb3f76ea34e47753e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46675 } }
I20260928 08:47:58.878158 16486 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8c5003d43984eefb3f76ea34e47753e, State: Initialized, Role: FOLLOWER
I20260928 08:47:58.878866 16486 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c5003d43984eefb3f76ea34e47753e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46675 } }
I20260928 08:47:58.879127 16486 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:47:58.879268 16486 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:47:58.879446 16486 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [term 1 FOLLOWER]: Advancing to term 2
I20260928 08:47:58.883652 16486 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c5003d43984eefb3f76ea34e47753e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46675 } }
I20260928 08:47:58.884238 16486 leader_election.cc:302] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8c5003d43984eefb3f76ea34e47753e; no voters:
I20260928 08:47:58.884918 16486 leader_election.cc:288] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 08:47:58.885063 16490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [term 2 FOLLOWER]: Leader election won for term 2
I20260928 08:47:58.886216 16490 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [term 2 LEADER]: Becoming Leader. State: Replica: d8c5003d43984eefb3f76ea34e47753e, State: Running, Role: LEADER
I20260928 08:47:58.886951 16490 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c5003d43984eefb3f76ea34e47753e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46675 } }
I20260928 08:47:58.887341 16486 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:47:58.889921 16492 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8c5003d43984eefb3f76ea34e47753e. Latest consensus state: current_term: 2 leader_uuid: "d8c5003d43984eefb3f76ea34e47753e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c5003d43984eefb3f76ea34e47753e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46675 } } }
I20260928 08:47:58.890460 16492 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [sys.catalog]: This master's current role is: LEADER
I20260928 08:47:58.891399 16491 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d8c5003d43984eefb3f76ea34e47753e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c5003d43984eefb3f76ea34e47753e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46675 } } }
I20260928 08:47:58.891882 16491 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [sys.catalog]: This master's current role is: LEADER
I20260928 08:47:58.894074 16495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:47:58.904944 16495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:47:58.907083 16495 catalog_manager.cc:1259] Loaded cluster ID: 1fe21506ceb94a12898c2c5747916494
I20260928 08:47:58.907297 16495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:47:58.913103 16495 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:47:58.915494 16495 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e: Loaded TSK: 0
I20260928 08:47:58.959519 16507 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 08:47:58.961926 16495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e: Generated new TSK 1
I20260928 08:47:58.962396 16495 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 08:47:59.211510 16488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:47:59.211879 16488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:47:59.212064 16488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:47:59.219645 16488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:47:59.219805 16488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:47:59.220075 16488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:47:59.230716 16488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:42723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36359
--tserver_master_addrs=127.0.97.62:46675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:47:59.232167 16488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:47:59.233531 16488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:47:59.249580 16514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:59.250026 16513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:59.249598 16516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:47:59.250445 16488 server_base.cc:1034] running on GCE node
I20260928 08:47:59.251322 16488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:47:59.252638 16488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:47:59.253909 16488 hybrid_clock.cc:648] HybridClock initialized: now 1790585279253850 us; error 43 us; skew 500 ppm
I20260928 08:47:59.254317 16488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:47:59 dist-test-slave-n6xs krb5kdc[15867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:47:59.258520 16488 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:47:59.260470 16488 webserver.cc:460] Webserver started at http://127.0.97.1:36359/ using document root <none> and password file <none>
I20260928 08:47:59.261446 16488 fs_manager.cc:359] Metadata directory not provided
I20260928 08:47:59.261607 16488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:47:59.268796 16488 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 08:47:59.272307 16523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:59.274031 16488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:47:59.274211 16488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/wal
uuid: "16395cd18c07415489f77aa172a80133"
format_stamp: "Formatted at 2026-09-28 08:47:36 on dist-test-slave-n6xs"
server_key: "29ff7f520798481cab12b239ba5f69d0"
server_key_iv: "ea648a53b3fcd812f1fe8b5425aab000"
server_key_version: "encryptionkey@0"
I20260928 08:47:59.275558 16488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:47:59.310585 16488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:47:59.316196 16488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:47:59.316489 16488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:47:59.317706 16488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:47:59.319442 16488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:47:59.319561 16488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:59.319664 16488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:47:59.319736 16488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:59.354677 16488 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42723
I20260928 08:47:59.354789 16635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42723 every 8 connection(s)
I20260928 08:47:59.356263 16488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:47:59.358349 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 16488
I20260928 08:47:59.359016 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:45079
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46675
--webserver_port=40311
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:47:59 dist-test-slave-n6xs krb5kdc[15867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:47:59.384884 16638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:59.359491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:49553 (local address 127.0.97.62:46675)
0928 08:47:59.360252 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:59.360274 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:59.360949 (+ 675us) server_negotiation.cc:407] Connection header received
0928 08:47:59.364087 (+ 3138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:59.364100 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:59.364477 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:59.364778 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:59.366891 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:59.368273 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:59.370872 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:59.371537 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:59.371701 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:59.376325 (+ 4624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:59.376334 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:59.376350 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:59.376389 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:59.379718 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:59.380928 (+ 1210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:59.380935 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:59.380941 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:59.381249 (+ 308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:59.381781 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:59.381785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:59.381786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:59.382361 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:59.382504 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:59.383540 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0928 08:47:59.383933 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":288,"threads_started":1}
I20260928 08:47:59.385476 16639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:59.359729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.1:49553)
0928 08:47:59.360716 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:59.360792 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:59.363494 (+ 2702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:59.364977 (+ 1483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:59.364991 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:59.365823 (+ 832us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:47:59.366635 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:59.366663 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:59.368457 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:59.368465 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:59.370687 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:59.370705 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:59.371801 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:59.371835 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:59.372141 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:59.372889 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:47:59.372923 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:47:59.376023 (+ 3100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:47:59.379924 (+ 3901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:59.379938 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:59.379960 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:59.380741 (+ 781us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:59.381383 (+ 642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:59.381391 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:59.381397 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:59.381658 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:59.382504 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:47:59.382514 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:47:59.383013 (+ 499us) client_negotiation.cc:769] Sending connection context
0928 08:47:59.383362 (+ 349us) client_negotiation.cc:240] Negotiation successful
0928 08:47:59.384042 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":385,"threads_started":1}
I20260928 08:47:59.388640 16636 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46675
I20260928 08:47:59.388988 16636 heartbeater.cc:464] Registering TS with master...
I20260928 08:47:59.389782 16636 heartbeater.cc:511] Master 127.0.97.62:46675 requested a full tablet report, sending...
I20260928 08:47:59.392215 16451 ts_manager.cc:194] Registered new tserver with Master: 16395cd18c07415489f77aa172a80133 (127.0.97.1:42723)
I20260928 08:47:59.394408 16451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:49553
W20260928 08:47:59.701924 16640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:47:59.702215 16640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:47:59.702308 16640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:47:59.709436 16640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:47:59.709554 16640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:47:59.709669 16640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:47:59.719357 16640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:45079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=40311
--tserver_master_addrs=127.0.97.62:46675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:47:59.720628 16640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:47:59.721872 16640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:47:59.735266 16645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:59.735229 16648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:47:59.735288 16646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:47:59.736682 16640 server_base.cc:1034] running on GCE node
I20260928 08:47:59.737274 16640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:47:59.738297 16640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:47:59.739493 16640 hybrid_clock.cc:648] HybridClock initialized: now 1790585279739427 us; error 59 us; skew 500 ppm
I20260928 08:47:59.739784 16640 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:47:59 dist-test-slave-n6xs krb5kdc[15867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:47:59.743166 16640 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:47:59.744733 16640 webserver.cc:460] Webserver started at http://127.0.97.2:40311/ using document root <none> and password file <none>
I20260928 08:47:59.745416 16640 fs_manager.cc:359] Metadata directory not provided
I20260928 08:47:59.745556 16640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:47:59.751114 16640 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 08:47:59.753938 16655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:59.755390 16640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 08:47:59.755584 16640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/wal
uuid: "c018f42c29c74413abcfd6054fcb9be2"
format_stamp: "Formatted at 2026-09-28 08:47:36 on dist-test-slave-n6xs"
server_key: "e7acf1de1fa2ce7210205ae55746084e"
server_key_iv: "f6d85eabd0984b921b082970a1b2bf03"
server_key_version: "encryptionkey@0"
I20260928 08:47:59.756734 16640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:47:59.774326 16640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:47:59.778947 16640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:47:59.779241 16640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:47:59.780421 16640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:47:59.781998 16640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:47:59.782117 16640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:59.782259 16640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:47:59.782347 16640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:47:59.819573 16640 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45079
I20260928 08:47:59.819657 16767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45079 every 8 connection(s)
I20260928 08:47:59.821447 16640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:47:59.827656 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 16640
I20260928 08:47:59.828378 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:42469
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46675
--webserver_port=37671
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 08:47:59 dist-test-slave-n6xs krb5kdc[15867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585279, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:47:59.844889 16638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:59.824205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:34055 (local address 127.0.97.62:46675)
0928 08:47:59.824416 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 08:47:59.824420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:47:59.825692 (+ 1272us) server_negotiation.cc:407] Connection header received
0928 08:47:59.827717 (+ 2025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:47:59.827724 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:47:59.827869 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:47:59.827971 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:47:59.830159 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:59.831299 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:59.833645 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:47:59.834454 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:47:59.834613 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:59.838929 (+ 4316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:47:59.838935 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:47:59.838938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:47:59.838972 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:47:59.841757 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:59.842473 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:59.842482 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:59.842486 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:59.842620 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:47:59.843053 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:47:59.843059 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:47:59.843062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:47:59.843378 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:47:59.843541 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 08:47:59.844291 (+ 750us) server_negotiation.cc:299] Negotiation successful
0928 08:47:59.844546 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 08:47:59.845974 16770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:47:59.824765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.2:34055)
0928 08:47:59.825524 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0928 08:47:59.825569 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 08:47:59.827364 (+ 1795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:47:59.828380 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:47:59.828392 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:47:59.829156 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:47:59.829932 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:59.829956 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:59.831482 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:59.831487 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:59.833481 (+ 1994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:47:59.833495 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:47:59.834697 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:47:59.834703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:47:59.834977 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:47:59.835850 (+ 873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:47:59.835881 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:47:59.838582 (+ 2701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:47:59.841909 (+ 3327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:59.841918 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:59.841936 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:59.842349 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:59.842737 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:47:59.842744 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:47:59.842748 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:47:59.842940 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:47:59.843506 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:47:59.843515 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:47:59.843932 (+ 417us) client_negotiation.cc:769] Sending connection context
0928 08:47:59.844203 (+ 271us) client_negotiation.cc:240] Negotiation successful
0928 08:47:59.844755 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":323,"threads_started":1}
I20260928 08:47:59.847560 16768 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46675
I20260928 08:47:59.847944 16768 heartbeater.cc:464] Registering TS with master...
I20260928 08:47:59.848685 16768 heartbeater.cc:511] Master 127.0.97.62:46675 requested a full tablet report, sending...
I20260928 08:47:59.850294 16451 ts_manager.cc:194] Registered new tserver with Master: c018f42c29c74413abcfd6054fcb9be2 (127.0.97.2:45079)
I20260928 08:47:59.851620 16451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:34055
W20260928 08:48:00.168474 16771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:48:00.168766 16771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:48:00.168861 16771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:48:00.176033 16771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:48:00.176158 16771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:48:00.176275 16771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:48:00.186852 16771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:42469
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=37671
--tserver_master_addrs=127.0.97.62:46675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:48:00.188315 16771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:48:00.189651 16771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:48:00.202878 16779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:00.202951 16776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:00.202922 16777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:48:00.204151 16771 server_base.cc:1034] running on GCE node
I20260928 08:48:00.204850 16771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:48:00.205847 16771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:48:00.207054 16771 hybrid_clock.cc:648] HybridClock initialized: now 1790585280206986 us; error 58 us; skew 500 ppm
I20260928 08:48:00.207396 16771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:48:00 dist-test-slave-n6xs krb5kdc[15867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:48:00.210558 16771 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:48:00.212049 16771 webserver.cc:460] Webserver started at http://127.0.97.3:37671/ using document root <none> and password file <none>
I20260928 08:48:00.212791 16771 fs_manager.cc:359] Metadata directory not provided
I20260928 08:48:00.212944 16771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:48:00.218551 16771 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260928 08:48:00.221407 16786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:00.222760 16771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:48:00.222936 16771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/wal
uuid: "59dc27f1dc6847b5acee11d95e81ba0f"
format_stamp: "Formatted at 2026-09-28 08:47:37 on dist-test-slave-n6xs"
server_key: "b45b76e3326428c205094c335aee1f6a"
server_key_iv: "b886b98f904466ac8534631e81540f77"
server_key_version: "encryptionkey@0"
I20260928 08:48:00.223984 16771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:48:00.237548 16771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:48:00.241573 16771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:48:00.241858 16771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:48:00.242969 16771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:48:00.244359 16771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:48:00.244444 16771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:00.244520 16771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:48:00.244577 16771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:00.277949 16771 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42469
I20260928 08:48:00.278041 16898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42469 every 8 connection(s)
I20260928 08:48:00.279563 16771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:48:00.286906 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 16771
Sep 28 08:48:00 dist-test-slave-n6xs krb5kdc[15867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585280, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:48:00.304416 16638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:00.282606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:44903 (local address 127.0.97.62:46675)
0928 08:48:00.282799 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:00.282804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:00.284144 (+ 1340us) server_negotiation.cc:407] Connection header received
0928 08:48:00.286028 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:00.286032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:00.286098 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:00.286230 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:00.288483 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:00.289734 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:00.292166 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:00.292884 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:00.293039 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:00.297500 (+ 4461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:00.297504 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:00.297508 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:48:00.297538 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:00.300792 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:00.301730 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:00.301734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:00.301736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:00.301845 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:00.302352 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:00.302355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:00.302357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:00.302623 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:00.302736 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:00.303589 (+ 853us) server_negotiation.cc:299] Negotiation successful
0928 08:48:00.304109 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 08:48:00.305550 16901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:00.283077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.3:44903)
0928 08:48:00.283959 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:00.284012 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:00.285768 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:00.286491 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:00.286505 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:00.287356 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:48:00.288227 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:00.288262 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:00.289914 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:00.289921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:00.292031 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:00.292043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:00.293194 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:00.293205 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:00.293487 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:00.294288 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:48:00.294315 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:48:00.297260 (+ 2945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:48:00.300946 (+ 3686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:48:00.300957 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:48:00.300976 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:48:00.301605 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:48:00.301962 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:48:00.301971 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:48:00.301975 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:48:00.302240 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:48:00.302755 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:48:00.302762 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:48:00.303112 (+ 350us) client_negotiation.cc:769] Sending connection context
0928 08:48:00.303453 (+ 341us) client_negotiation.cc:240] Negotiation successful
0928 08:48:00.304050 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":323,"threads_started":1}
I20260928 08:48:00.305613 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:00.288053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49282 (local address 127.0.97.62:46675)
0928 08:48:00.289353 (+ 1300us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:00.289360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:00.289380 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:48:00.289436 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:00.289441 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:00.289520 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:00.289644 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:00.291750 (+ 2106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:00.292902 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:00.294898 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:00.295321 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:00.295422 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:00.298790 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:00.298795 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:00.298798 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:48:00.298855 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:00.302231 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:00.303287 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:00.303293 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:00.303296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:00.303453 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:00.303916 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:00.303923 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:00.303926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:00.304315 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:00.304601 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:00.304962 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 08:48:00.305192 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1172,"thread_start_us":1006,"threads_started":1}
I20260928 08:48:00.307358 16899 heartbeater.cc:347] Connected to a master server at 127.0.97.62:46675
I20260928 08:48:00.307715 16899 heartbeater.cc:464] Registering TS with master...
I20260928 08:48:00.308548 16899 heartbeater.cc:511] Master 127.0.97.62:46675 requested a full tablet report, sending...
I20260928 08:48:00.309870 16451 ts_manager.cc:194] Registered new tserver with Master: 59dc27f1dc6847b5acee11d95e81ba0f (127.0.97.3:42469)
I20260928 08:48:00.310734 16451 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:44903
I20260928 08:48:00.320973 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:48:00.327574 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:00.321939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49294 (local address 127.0.97.62:46675)
0928 08:48:00.322143 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:00.322148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:00.322619 (+ 471us) server_negotiation.cc:407] Connection header received
0928 08:48:00.322797 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:00.322801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:00.322928 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:00.323114 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:48:00.323503 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:00.324305 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:00.325766 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:00.326172 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:00.326277 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:00.326744 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:48:00.326980 (+ 236us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 08:48:00.327329 (+ 349us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49294: token expired
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 08:48:00.327672 16903 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49294: token expired
W20260928 08:48:00.328265 16374 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.97.62:46675)
I20260928 08:48:00.341153 16903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:00.328719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49306 (local address 127.0.97.62:46675)
0928 08:48:00.328870 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:00.328873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:00.328903 (+ 30us) server_negotiation.cc:407] Connection header received
0928 08:48:00.329080 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:00.329084 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:00.329139 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:00.329214 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:00.330609 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:00.331278 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:00.332655 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:00.333027 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:00.333136 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:00.335708 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:00.335712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:00.335716 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:48:00.335754 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:00.338477 (+ 2723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:00.339186 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:00.339190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:00.339192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:00.339313 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:00.339714 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:00.339719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:00.339721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:00.340195 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:00.340404 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:00.340672 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 08:48:00.340845 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 08:48:00.344937 16451 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:48:00.353395 16451 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:48:00.377878 16909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:00.366311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:42723 (local address 127.0.0.1:45346)
0928 08:48:00.367428 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:00.367477 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:00.367683 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:00.369593 (+ 1910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:00.369606 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:00.369643 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:48:00.370121 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:00.370133 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:00.372359 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:00.372366 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:00.374858 (+ 2492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:00.374878 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:00.376847 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:00.376853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:00.377057 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:00.377125 (+ 68us) client_negotiation.cc:769] Sending connection context
0928 08:48:00.377233 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 08:48:00.377431 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"mutex_wait_us":1,"thread_start_us":558,"threads_started":1}
I20260928 08:48:00.379128 16912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:00.366988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57008 (local address 127.0.97.2:45079)
0928 08:48:00.368445 (+ 1457us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:00.368458 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:00.369930 (+ 1472us) server_negotiation.cc:407] Connection header received
0928 08:48:00.370257 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:00.370268 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:00.370497 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:00.370677 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:48:00.371489 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:00.373031 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:00.376257 (+ 3226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:00.377674 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:00.377766 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:00.377985 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:00.378592 (+ 607us) server_negotiation.cc:299] Negotiation successful
0928 08:48:00.378833 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1241,"thread_start_us":1119,"threads_started":1}
I20260928 08:48:00.379755 16911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:00.366606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45346 (local address 127.0.97.1:42723)
0928 08:48:00.368918 (+ 2312us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:00.368926 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:00.368952 (+ 26us) server_negotiation.cc:407] Connection header received
0928 08:48:00.369044 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:00.369052 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:00.369320 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:00.370327 (+ 1007us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:48:00.370548 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:00.372211 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:00.375045 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:00.376608 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:00.378679 (+ 2071us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:00.378978 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:00.379043 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 08:48:00.379433 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2165,"thread_start_us":2024,"threads_started":1}
I20260928 08:48:00.379783 16910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:00.366520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45079 (local address 127.0.0.1:57008)
0928 08:48:00.369819 (+ 3299us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:00.369843 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:00.370043 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:00.370934 (+ 891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:00.370938 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:00.370963 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:48:00.371325 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:00.371335 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:00.373214 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:00.373219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:00.376057 (+ 2838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:00.376073 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:00.378192 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:00.378201 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:00.378378 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:00.378423 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 08:48:00.379328 (+ 905us) client_negotiation.cc:240] Negotiation successful
0928 08:48:00.379455 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3142,"thread_start_us":2990,"threads_started":1}
I20260928 08:48:00.384133 16571 tablet_service.cc:1467] Processing CreateTablet for tablet 4bacf3287817431fa6a2761444c7fb07 (DEFAULT_TABLE table=test-table [id=a17dfa8e24d843b4a94ce506ff4db13a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:48:00.384261 16703 tablet_service.cc:1467] Processing CreateTablet for tablet 4bacf3287817431fa6a2761444c7fb07 (DEFAULT_TABLE table=test-table [id=a17dfa8e24d843b4a94ce506ff4db13a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:48:00.385838 16571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bacf3287817431fa6a2761444c7fb07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:48:00.386214 16703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bacf3287817431fa6a2761444c7fb07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:48:00.388130 16909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:00.369657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42469 (local address 127.0.0.1:36756)
0928 08:48:00.378156 (+ 8499us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:00.378179 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:00.378266 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:00.381156 (+ 2890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:00.381161 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:00.381187 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:48:00.381370 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:00.381380 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:00.383685 (+ 2305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:00.383690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:00.385534 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:00.385548 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:00.387278 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:00.387284 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:00.387493 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:00.387534 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 08:48:00.387614 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 08:48:00.387754 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8376,"mutex_wait_us":225,"threads_started":1}
I20260928 08:48:00.389096 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:00.370037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36756 (local address 127.0.97.3:42469)
0928 08:48:00.380413 (+ 10376us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:00.380423 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:00.380459 (+ 36us) server_negotiation.cc:407] Connection header received
0928 08:48:00.380556 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:00.380576 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:00.380896 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:00.381556 (+ 660us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:48:00.381907 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:00.383519 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:00.385751 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:00.387099 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:00.388325 (+ 1226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:00.388553 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:00.388641 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 08:48:00.388801 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10173,"threads_started":1}
I20260928 08:48:00.397812 16636 heartbeater.cc:503] Master 127.0.97.62:46675 was elected leader, sending a full tablet report...
I20260928 08:48:00.408631 16915 tablet_bootstrap.cc:492] T 4bacf3287817431fa6a2761444c7fb07 P 16395cd18c07415489f77aa172a80133: Bootstrap starting.
I20260928 08:48:00.413326 16916 tablet_bootstrap.cc:492] T 4bacf3287817431fa6a2761444c7fb07 P c018f42c29c74413abcfd6054fcb9be2: Bootstrap starting.
I20260928 08:48:00.413790 16915 tablet_bootstrap.cc:654] T 4bacf3287817431fa6a2761444c7fb07 P 16395cd18c07415489f77aa172a80133: Neither blocks nor log segments found. Creating new log.
I20260928 08:48:00.413215 16834 tablet_service.cc:1467] Processing CreateTablet for tablet 4bacf3287817431fa6a2761444c7fb07 (DEFAULT_TABLE table=test-table [id=a17dfa8e24d843b4a94ce506ff4db13a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:48:00.415033 16834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bacf3287817431fa6a2761444c7fb07. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:48:00.415020 16915 log.cc:824] T 4bacf3287817431fa6a2761444c7fb07 P 16395cd18c07415489f77aa172a80133: Log is configured to *not* fsync() on all Append() calls
I20260928 08:48:00.417297 16916 tablet_bootstrap.cc:654] T 4bacf3287817431fa6a2761444c7fb07 P c018f42c29c74413abcfd6054fcb9be2: Neither blocks nor log segments found. Creating new log.
I20260928 08:48:00.418427 16915 tablet_bootstrap.cc:492] T 4bacf3287817431fa6a2761444c7fb07 P 16395cd18c07415489f77aa172a80133: No bootstrap required, opened a new log
I20260928 08:48:00.418530 16916 log.cc:824] T 4bacf3287817431fa6a2761444c7fb07 P c018f42c29c74413abcfd6054fcb9be2: Log is configured to *not* fsync() on all Append() calls
I20260928 08:48:00.418823 16915 ts_tablet_manager.cc:1397] T 4bacf3287817431fa6a2761444c7fb07 P 16395cd18c07415489f77aa172a80133: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260928 08:48:00.422395 16916 tablet_bootstrap.cc:492] T 4bacf3287817431fa6a2761444c7fb07 P c018f42c29c74413abcfd6054fcb9be2: No bootstrap required, opened a new log
I20260928 08:48:00.422668 16916 ts_tablet_manager.cc:1397] T 4bacf3287817431fa6a2761444c7fb07 P c018f42c29c74413abcfd6054fcb9be2: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 08:48:00.426730 16915 raft_consensus.cc:348] T 4bacf3287817431fa6a2761444c7fb07 P 16395cd18c07415489f77aa172a80133 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16395cd18c07415489f77aa172a80133" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42723 } } peers { permanent_uuid: "c018f42c29c74413abcfd6054fcb9be2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45079 } } peers { permanent_uuid: "59dc27f1dc6847b5acee11d95e81ba0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42469 } }
I20260928 08:48:00.427122 16915 raft_consensus.cc:374] T 4bacf3287817431fa6a2761444c7fb07 P 16395cd18c07415489f77aa172a80133 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:48:00.427251 16915 raft_consensus.cc:729] T 4bacf3287817431fa6a2761444c7fb07 P 16395cd18c07415489f77aa172a80133 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16395cd18c07415489f77aa172a80133, State: Initialized, Role: FOLLOWER
I20260928 08:48:00.427976 16915 consensus_queue.cc:260] T 4bacf3287817431fa6a2761444c7fb07 P 16395cd18c07415489f77aa172a80133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16395cd18c07415489f77aa172a80133" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42723 } } peers { permanent_uuid: "c018f42c29c74413abcfd6054fcb9be2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45079 } } peers { permanent_uuid: "59dc27f1dc6847b5acee11d95e81ba0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42469 } }
I20260928 08:48:00.431260 16920 tablet_bootstrap.cc:492] T 4bacf3287817431fa6a2761444c7fb07 P 59dc27f1dc6847b5acee11d95e81ba0f: Bootstrap starting.
I20260928 08:48:00.432955 16916 raft_consensus.cc:348] T 4bacf3287817431fa6a2761444c7fb07 P c018f42c29c74413abcfd6054fcb9be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16395cd18c07415489f77aa172a80133" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42723 } } peers { permanent_uuid: "c018f42c29c74413abcfd6054fcb9be2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45079 } } peers { permanent_uuid: "59dc27f1dc6847b5acee11d95e81ba0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42469 } }
I20260928 08:48:00.433336 16916 raft_consensus.cc:374] T 4bacf3287817431fa6a2761444c7fb07 P c018f42c29c74413abcfd6054fcb9be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:48:00.433455 16916 raft_consensus.cc:729] T 4bacf3287817431fa6a2761444c7fb07 P c018f42c29c74413abcfd6054fcb9be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c018f42c29c74413abcfd6054fcb9be2, State: Initialized, Role: FOLLOWER
I20260928 08:48:00.434826 16915 ts_tablet_manager.cc:1428] T 4bacf3287817431fa6a2761444c7fb07 P 16395cd18c07415489f77aa172a80133: Time spent starting tablet: real 0.016s user 0.012s sys 0.003s
I20260928 08:48:00.434216 16916 consensus_queue.cc:260] T 4bacf3287817431fa6a2761444c7fb07 P c018f42c29c74413abcfd6054fcb9be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16395cd18c07415489f77aa172a80133" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42723 } } peers { permanent_uuid: "c018f42c29c74413abcfd6054fcb9be2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45079 } } peers { permanent_uuid: "59dc27f1dc6847b5acee11d95e81ba0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42469 } }
I20260928 08:48:00.436141 16768 heartbeater.cc:503] Master 127.0.97.62:46675 was elected leader, sending a full tablet report...
I20260928 08:48:00.436489 16920 tablet_bootstrap.cc:654] T 4bacf3287817431fa6a2761444c7fb07 P 59dc27f1dc6847b5acee11d95e81ba0f: Neither blocks nor log segments found. Creating new log.
I20260928 08:48:00.436785 16916 ts_tablet_manager.cc:1428] T 4bacf3287817431fa6a2761444c7fb07 P c018f42c29c74413abcfd6054fcb9be2: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260928 08:48:00.437844 16920 log.cc:824] T 4bacf3287817431fa6a2761444c7fb07 P 59dc27f1dc6847b5acee11d95e81ba0f: Log is configured to *not* fsync() on all Append() calls
I20260928 08:48:00.440937 16920 tablet_bootstrap.cc:492] T 4bacf3287817431fa6a2761444c7fb07 P 59dc27f1dc6847b5acee11d95e81ba0f: No bootstrap required, opened a new log
I20260928 08:48:00.441231 16920 ts_tablet_manager.cc:1397] T 4bacf3287817431fa6a2761444c7fb07 P 59dc27f1dc6847b5acee11d95e81ba0f: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 08:48:00.448203 16920 raft_consensus.cc:348] T 4bacf3287817431fa6a2761444c7fb07 P 59dc27f1dc6847b5acee11d95e81ba0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16395cd18c07415489f77aa172a80133" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42723 } } peers { permanent_uuid: "c018f42c29c74413abcfd6054fcb9be2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45079 } } peers { permanent_uuid: "59dc27f1dc6847b5acee11d95e81ba0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42469 } }
I20260928 08:48:00.448611 16920 raft_consensus.cc:374] T 4bacf3287817431fa6a2761444c7fb07 P 59dc27f1dc6847b5acee11d95e81ba0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:48:00.448719 16920 raft_consensus.cc:729] T 4bacf3287817431fa6a2761444c7fb07 P 59dc27f1dc6847b5acee11d95e81ba0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59dc27f1dc6847b5acee11d95e81ba0f, State: Initialized, Role: FOLLOWER
I20260928 08:48:00.449296 16920 consensus_queue.cc:260] T 4bacf3287817431fa6a2761444c7fb07 P 59dc27f1dc6847b5acee11d95e81ba0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16395cd18c07415489f77aa172a80133" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42723 } } peers { permanent_uuid: "c018f42c29c74413abcfd6054fcb9be2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45079 } } peers { permanent_uuid: "59dc27f1dc6847b5acee11d95e81ba0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42469 } }
I20260928 08:48:00.450425 16899 heartbeater.cc:503] Master 127.0.97.62:46675 was elected leader, sending a full tablet report...
I20260928 08:48:00.451231 16920 ts_tablet_manager.cc:1428] T 4bacf3287817431fa6a2761444c7fb07 P 59dc27f1dc6847b5acee11d95e81ba0f: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
W20260928 08:48:00.531443 16900 tablet.cc:2369] T 4bacf3287817431fa6a2761444c7fb07 P 59dc27f1dc6847b5acee11d95e81ba0f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:48:00.573545 16769 tablet.cc:2369] T 4bacf3287817431fa6a2761444c7fb07 P c018f42c29c74413abcfd6054fcb9be2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:48:00.608741 16637 tablet.cc:2369] T 4bacf3287817431fa6a2761444c7fb07 P 16395cd18c07415489f77aa172a80133: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:48:00.732954 16919 raft_consensus.cc:482] T 4bacf3287817431fa6a2761444c7fb07 P 16395cd18c07415489f77aa172a80133 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:48:00.733330 16919 raft_consensus.cc:504] T 4bacf3287817431fa6a2761444c7fb07 P 16395cd18c07415489f77aa172a80133 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16395cd18c07415489f77aa172a80133" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42723 } } peers { permanent_uuid: "c018f42c29c74413abcfd6054fcb9be2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45079 } } peers { permanent_uuid: "59dc27f1dc6847b5acee11d95e81ba0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42469 } }
I20260928 08:48:00.735095 16919 leader_election.cc:288] T 4bacf3287817431fa6a2761444c7fb07 P 16395cd18c07415489f77aa172a80133 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c018f42c29c74413abcfd6054fcb9be2 (127.0.97.2:45079), 59dc27f1dc6847b5acee11d95e81ba0f (127.0.97.3:42469)
I20260928 08:48:00.742276 16912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:00.735452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60963 (local address 127.0.97.2:45079)
0928 08:48:00.735724 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:00.735730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:00.736538 (+ 808us) server_negotiation.cc:407] Connection header received
0928 08:48:00.736774 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:00.736779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:00.736860 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:00.736937 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:48:00.737842 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:00.738594 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:00.740539 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:00.741313 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:00.741475 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:00.741578 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:00.741895 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 08:48:00.742014 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260928 08:48:00.742391 16924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:00.735394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:45079 (local address 127.0.97.1:60963)
0928 08:48:00.736411 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:00.736427 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:00.736566 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:00.737042 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:00.737048 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:00.737091 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:48:00.737662 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:00.737674 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:00.738735 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:00.738740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:00.740384 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:00.740396 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:00.741547 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:00.741553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:00.741708 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:00.741733 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 08:48:00.741857 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 08:48:00.742000 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"mutex_wait_us":23,"thread_start_us":502,"threads_started":1}
I20260928 08:48:00.742502 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:00.735615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:59247 (local address 127.0.97.3:42469)
0928 08:48:00.735836 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:00.735846 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:00.736407 (+ 561us) server_negotiation.cc:407] Connection header received
0928 08:48:00.736666 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:00.736670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:00.736765 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:00.736855 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:48:00.737761 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:00.738639 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:00.740620 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:00.741422 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:00.741610 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:00.741738 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:00.742093 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 08:48:00.742224 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 08:48:00.742960 16925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:00.735334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42469 (local address 127.0.97.1:59247)
0928 08:48:00.736310 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:00.736330 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:00.736510 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:00.736892 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:00.736896 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:00.736921 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:48:00.737567 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:00.737585 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:00.738794 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:00.738807 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:00.740457 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:00.740470 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:00.741688 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:00.741694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:00.741872 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:00.741909 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 08:48:00.742055 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 08:48:00.742191 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"mutex_wait_us":21,"spinlock_wait_cycles":3840,"thread_start_us":887,"threads_started":1}
I20260928 08:48:00.743637 16723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bacf3287817431fa6a2761444c7fb07" candidate_uuid: "16395cd18c07415489f77aa172a80133" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c018f42c29c74413abcfd6054fcb9be2" is_pre_election: true
I20260928 08:48:00.743907 16854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bacf3287817431fa6a2761444c7fb07" candidate_uuid: "16395cd18c07415489f77aa172a80133" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59dc27f1dc6847b5acee11d95e81ba0f" is_pre_election: true
I20260928 08:48:00.744277 16723 raft_consensus.cc:2454] T 4bacf3287817431fa6a2761444c7fb07 P c018f42c29c74413abcfd6054fcb9be2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16395cd18c07415489f77aa172a80133 in term 0.
I20260928 08:48:00.744397 16854 raft_consensus.cc:2454] T 4bacf3287817431fa6a2761444c7fb07 P 59dc27f1dc6847b5acee11d95e81ba0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16395cd18c07415489f77aa172a80133 in term 0.
I20260928 08:48:00.745097 16526 leader_election.cc:302] T 4bacf3287817431fa6a2761444c7fb07 P 16395cd18c07415489f77aa172a80133 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16395cd18c07415489f77aa172a80133, 59dc27f1dc6847b5acee11d95e81ba0f; no voters:
I20260928 08:48:00.745569 16919 raft_consensus.cc:2781] T 4bacf3287817431fa6a2761444c7fb07 P 16395cd18c07415489f77aa172a80133 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:48:00.745728 16919 raft_consensus.cc:482] T 4bacf3287817431fa6a2761444c7fb07 P 16395cd18c07415489f77aa172a80133 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:48:00.745846 16919 raft_consensus.cc:3037] T 4bacf3287817431fa6a2761444c7fb07 P 16395cd18c07415489f77aa172a80133 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:48:00.748344 16919 raft_consensus.cc:504] T 4bacf3287817431fa6a2761444c7fb07 P 16395cd18c07415489f77aa172a80133 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16395cd18c07415489f77aa172a80133" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42723 } } peers { permanent_uuid: "c018f42c29c74413abcfd6054fcb9be2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45079 } } peers { permanent_uuid: "59dc27f1dc6847b5acee11d95e81ba0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42469 } }
I20260928 08:48:00.749020 16919 leader_election.cc:288] T 4bacf3287817431fa6a2761444c7fb07 P 16395cd18c07415489f77aa172a80133 [CANDIDATE]: Term 1 election: Requested vote from peers c018f42c29c74413abcfd6054fcb9be2 (127.0.97.2:45079), 59dc27f1dc6847b5acee11d95e81ba0f (127.0.97.3:42469)
I20260928 08:48:00.749562 16723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bacf3287817431fa6a2761444c7fb07" candidate_uuid: "16395cd18c07415489f77aa172a80133" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c018f42c29c74413abcfd6054fcb9be2"
I20260928 08:48:00.749552 16854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bacf3287817431fa6a2761444c7fb07" candidate_uuid: "16395cd18c07415489f77aa172a80133" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59dc27f1dc6847b5acee11d95e81ba0f"
I20260928 08:48:00.749795 16723 raft_consensus.cc:3037] T 4bacf3287817431fa6a2761444c7fb07 P c018f42c29c74413abcfd6054fcb9be2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:48:00.749795 16854 raft_consensus.cc:3037] T 4bacf3287817431fa6a2761444c7fb07 P 59dc27f1dc6847b5acee11d95e81ba0f [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:48:00.753328 16723 raft_consensus.cc:2454] T 4bacf3287817431fa6a2761444c7fb07 P c018f42c29c74413abcfd6054fcb9be2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16395cd18c07415489f77aa172a80133 in term 1.
I20260928 08:48:00.753328 16854 raft_consensus.cc:2454] T 4bacf3287817431fa6a2761444c7fb07 P 59dc27f1dc6847b5acee11d95e81ba0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16395cd18c07415489f77aa172a80133 in term 1.
I20260928 08:48:00.753847 16526 leader_election.cc:302] T 4bacf3287817431fa6a2761444c7fb07 P 16395cd18c07415489f77aa172a80133 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16395cd18c07415489f77aa172a80133, 59dc27f1dc6847b5acee11d95e81ba0f; no voters:
I20260928 08:48:00.754197 16919 raft_consensus.cc:2781] T 4bacf3287817431fa6a2761444c7fb07 P 16395cd18c07415489f77aa172a80133 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:48:00.754685 16919 raft_consensus.cc:686] T 4bacf3287817431fa6a2761444c7fb07 P 16395cd18c07415489f77aa172a80133 [term 1 LEADER]: Becoming Leader. State: Replica: 16395cd18c07415489f77aa172a80133, State: Running, Role: LEADER
I20260928 08:48:00.755179 16919 consensus_queue.cc:237] T 4bacf3287817431fa6a2761444c7fb07 P 16395cd18c07415489f77aa172a80133 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16395cd18c07415489f77aa172a80133" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42723 } } peers { permanent_uuid: "c018f42c29c74413abcfd6054fcb9be2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45079 } } peers { permanent_uuid: "59dc27f1dc6847b5acee11d95e81ba0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42469 } }
I20260928 08:48:00.760294 16451 catalog_manager.cc:5495] T 4bacf3287817431fa6a2761444c7fb07 P 16395cd18c07415489f77aa172a80133 reported cstate change: term changed from 0 to 1, leader changed from <none> to 16395cd18c07415489f77aa172a80133 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "16395cd18c07415489f77aa172a80133" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16395cd18c07415489f77aa172a80133" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 42723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c018f42c29c74413abcfd6054fcb9be2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59dc27f1dc6847b5acee11d95e81ba0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42469 } health_report { overall_health: UNKNOWN } } }
I20260928 08:48:00.839087 16911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:00.833275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45354 (local address 127.0.97.1:42723)
0928 08:48:00.833472 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:00.833477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:00.833497 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:48:00.833609 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:00.833614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:00.833720 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:00.833883 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:48:00.834179 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:00.835274 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:00.836723 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:00.837340 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:00.837477 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:00.837954 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:48:00.838472 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:48:00.838613 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:00.838771 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 08:48:00.838845 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 08:48:00.847779 16723 raft_consensus.cc:1261] T 4bacf3287817431fa6a2761444c7fb07 P c018f42c29c74413abcfd6054fcb9be2 [term 1 FOLLOWER]: Refusing update from remote peer 16395cd18c07415489f77aa172a80133: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:48:00.847888 16854 raft_consensus.cc:1261] T 4bacf3287817431fa6a2761444c7fb07 P 59dc27f1dc6847b5acee11d95e81ba0f [term 1 FOLLOWER]: Refusing update from remote peer 16395cd18c07415489f77aa172a80133: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:48:00.849134 16919 consensus_queue.cc:1035] T 4bacf3287817431fa6a2761444c7fb07 P 16395cd18c07415489f77aa172a80133 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59dc27f1dc6847b5acee11d95e81ba0f" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:48:00.849489 16926 consensus_queue.cc:1035] T 4bacf3287817431fa6a2761444c7fb07 P 16395cd18c07415489f77aa172a80133 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c018f42c29c74413abcfd6054fcb9be2" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 45079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:48:00.866446 16931 mvcc.cc:205] Tried to move back new op lower bound from 7334237310341890048 to 7334237309977133056. Current Snapshot: MvccSnapshot[applied={T|T < 7334237310341890048}]
I20260928 08:48:00.874886 16451 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:48:00.875121 16451 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:48:00.878623 16451 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e: Sending DeleteTablet for 3 replicas of tablet 4bacf3287817431fa6a2761444c7fb07
I20260928 08:48:00.880023 16834 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bacf3287817431fa6a2761444c7fb07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:48:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:36756
I20260928 08:48:00.880070 16571 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bacf3287817431fa6a2761444c7fb07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:48:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:45346
I20260928 08:48:00.880165 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 16488
I20260928 08:48:00.880560 16703 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bacf3287817431fa6a2761444c7fb07 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:48:00 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57008
I20260928 08:48:00.889675 16941 tablet_replica.cc:330] T 4bacf3287817431fa6a2761444c7fb07 P c018f42c29c74413abcfd6054fcb9be2: stopping tablet replica
I20260928 08:48:00.889933 16940 tablet_replica.cc:330] T 4bacf3287817431fa6a2761444c7fb07 P 59dc27f1dc6847b5acee11d95e81ba0f: stopping tablet replica
I20260928 08:48:00.890201 16941 raft_consensus.cc:2229] T 4bacf3287817431fa6a2761444c7fb07 P c018f42c29c74413abcfd6054fcb9be2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:48:00.890455 16940 raft_consensus.cc:2229] T 4bacf3287817431fa6a2761444c7fb07 P 59dc27f1dc6847b5acee11d95e81ba0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:48:00.890794 16941 raft_consensus.cc:2258] T 4bacf3287817431fa6a2761444c7fb07 P c018f42c29c74413abcfd6054fcb9be2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:48:00.891006 16940 raft_consensus.cc:2258] T 4bacf3287817431fa6a2761444c7fb07 P 59dc27f1dc6847b5acee11d95e81ba0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:48:00.891744 16939 tablet_replica.cc:330] T 4bacf3287817431fa6a2761444c7fb07 P 16395cd18c07415489f77aa172a80133: stopping tablet replica
I20260928 08:48:00.892241 16939 raft_consensus.cc:2229] T 4bacf3287817431fa6a2761444c7fb07 P 16395cd18c07415489f77aa172a80133 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:48:00.892946 16939 raft_consensus.cc:2258] T 4bacf3287817431fa6a2761444c7fb07 P 16395cd18c07415489f77aa172a80133 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:48:00.894312 16941 ts_tablet_manager.cc:1905] T 4bacf3287817431fa6a2761444c7fb07 P c018f42c29c74413abcfd6054fcb9be2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:48:00.894312 16940 ts_tablet_manager.cc:1905] T 4bacf3287817431fa6a2761444c7fb07 P 59dc27f1dc6847b5acee11d95e81ba0f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:48:00.894989 16939 ts_tablet_manager.cc:1905] T 4bacf3287817431fa6a2761444c7fb07 P 16395cd18c07415489f77aa172a80133: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:48:00.901806 16940 ts_tablet_manager.cc:1918] T 4bacf3287817431fa6a2761444c7fb07 P 59dc27f1dc6847b5acee11d95e81ba0f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:48:00.902063 16940 log.cc:1196] T 4bacf3287817431fa6a2761444c7fb07 P 59dc27f1dc6847b5acee11d95e81ba0f: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/wal/wals/4bacf3287817431fa6a2761444c7fb07
I20260928 08:48:00.902329 16939 ts_tablet_manager.cc:1918] T 4bacf3287817431fa6a2761444c7fb07 P 16395cd18c07415489f77aa172a80133: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:48:00.902500 16939 log.cc:1196] T 4bacf3287817431fa6a2761444c7fb07 P 16395cd18c07415489f77aa172a80133: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/wal/wals/4bacf3287817431fa6a2761444c7fb07
I20260928 08:48:00.902881 16940 ts_tablet_manager.cc:1939] T 4bacf3287817431fa6a2761444c7fb07 P 59dc27f1dc6847b5acee11d95e81ba0f: Deleting consensus metadata
I20260928 08:48:00.903040 16939 ts_tablet_manager.cc:1939] T 4bacf3287817431fa6a2761444c7fb07 P 16395cd18c07415489f77aa172a80133: Deleting consensus metadata
I20260928 08:48:00.904754 16941 ts_tablet_manager.cc:1918] T 4bacf3287817431fa6a2761444c7fb07 P c018f42c29c74413abcfd6054fcb9be2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:48:00.904923 16941 log.cc:1196] T 4bacf3287817431fa6a2761444c7fb07 P c018f42c29c74413abcfd6054fcb9be2: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/wal/wals/4bacf3287817431fa6a2761444c7fb07
I20260928 08:48:00.905148 16436 catalog_manager.cc:4841] TS 16395cd18c07415489f77aa172a80133 (127.0.97.1:42723): tablet 4bacf3287817431fa6a2761444c7fb07 (table test-table [id=a17dfa8e24d843b4a94ce506ff4db13a]) successfully deleted
I20260928 08:48:00.905459 16941 ts_tablet_manager.cc:1939] T 4bacf3287817431fa6a2761444c7fb07 P c018f42c29c74413abcfd6054fcb9be2: Deleting consensus metadata
I20260928 08:48:00.905563 16437 catalog_manager.cc:4841] TS 59dc27f1dc6847b5acee11d95e81ba0f (127.0.97.3:42469): tablet 4bacf3287817431fa6a2761444c7fb07 (table test-table [id=a17dfa8e24d843b4a94ce506ff4db13a]) successfully deleted
I20260928 08:48:00.907078 16438 catalog_manager.cc:4841] TS c018f42c29c74413abcfd6054fcb9be2 (127.0.97.2:45079): tablet 4bacf3287817431fa6a2761444c7fb07 (table test-table [id=a17dfa8e24d843b4a94ce506ff4db13a]) successfully deleted
I20260928 08:48:00.918967 16911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:00.881336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45366 (local address 127.0.97.1:42723)
0928 08:48:00.881556 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:00.881566 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:00.897275 (+ 15709us) server_negotiation.cc:407] Connection header received
0928 08:48:00.897571 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:00.897578 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:00.897692 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:00.897769 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:00.904879 (+ 7110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:00.906144 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:00.908459 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:00.909007 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:00.909121 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:00.912730 (+ 3609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:00.912742 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:00.912758 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:48:00.912801 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:00.916001 (+ 3200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:00.916924 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:00.916932 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:00.916934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:00.917087 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:00.917509 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:00.917512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:00.917514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:00.917866 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:00.918230 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:00.918478 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 08:48:00.918670 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 08:48:00.919770 16631 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:48:01.020545 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 16488
I20260928 08:48:01.038702 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 16640
I20260928 08:48:01.052978 16912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:01.039440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57010 (local address 127.0.97.2:45079)
0928 08:48:01.039630 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:01.039636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:01.039733 (+ 97us) server_negotiation.cc:407] Connection header received
0928 08:48:01.040011 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:01.040017 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:01.040141 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:01.040256 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:01.041595 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:01.042306 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:01.043654 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:01.044239 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:01.044366 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:01.046967 (+ 2601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:01.046981 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:01.046997 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:48:01.047034 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:01.049881 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:01.050668 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:01.050681 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:01.050684 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:01.050815 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:01.051212 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:01.051218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:01.051221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:01.051613 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:01.051942 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:01.052362 (+ 420us) server_negotiation.cc:299] Negotiation successful
0928 08:48:01.052602 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 08:48:01.053637 16763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:48:01.144857 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 16640
I20260928 08:48:01.162137 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 16771
I20260928 08:48:01.179446 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:01.162958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36768 (local address 127.0.97.3:42469)
0928 08:48:01.163135 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:01.163140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:01.163167 (+ 27us) server_negotiation.cc:407] Connection header received
0928 08:48:01.163325 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:01.163329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:01.163394 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:01.163476 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:01.164867 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:01.165599 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:01.166956 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:01.167334 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:01.167425 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:01.173984 (+ 6559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:01.174001 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:01.174018 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:48:01.174064 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:01.176920 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:01.177589 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:01.177597 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:01.177599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:01.177741 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:01.178044 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:01.178047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:01.178049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:01.178402 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:01.178619 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:01.178867 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 08:48:01.179032 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 08:48:01.180245 16894 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:48:01.270607 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 16771
I20260928 08:48:01.287781 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 16418
I20260928 08:48:01.289243 16481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:48:01.404790 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 16418
I20260928 08:48:21.431042 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46675
--webserver_interface=127.0.97.62
--webserver_port=34085
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:46675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:48:21.769984 16947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:48:21.770277 16947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:48:21.770367 16947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:48:21.777683 16947 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 08:48:21.777793 16947 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 08:48:21.777832 16947 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 08:48:21.777878 16947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:48:21.777925 16947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:48:21.777987 16947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:48:21.778033 16947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:48:21.778080 16947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:48:21.787926 16947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:46675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:46675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=34085
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:48:21.789188 16947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:48:21.790475 16947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:48:21.803378 16952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:21.803476 16953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:21.803423 16955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:48:21.809569 16947 server_base.cc:1034] running on GCE node
I20260928 08:48:21.810461 16947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:48:21.812072 16947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:48:21.813299 16947 hybrid_clock.cc:648] HybridClock initialized: now 1790585301813231 us; error 40 us; skew 500 ppm
I20260928 08:48:21.813663 16947 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:48:21 dist-test-slave-n6xs krb5kdc[15867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585301, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:48:21.817158 16947 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:48:21.818818 16947 webserver.cc:460] Webserver started at http://127.0.97.62:34085/ using document root <none> and password file <none>
I20260928 08:48:21.819577 16947 fs_manager.cc:359] Metadata directory not provided
I20260928 08:48:21.819670 16947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:48:21.825567 16947 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260928 08:48:21.828595 16962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:21.830078 16947 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:48:21.830253 16947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "d8c5003d43984eefb3f76ea34e47753e"
format_stamp: "Formatted at 2026-09-28 08:47:35 on dist-test-slave-n6xs"
server_key: "be3ca84b3e8b3f4c29b0c28ec60c08e5"
server_key_iv: "b415d4a977937e6ad249e15fe290953b"
server_key_version: "encryptionkey@0"
I20260928 08:48:21.831213 16947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:48:21.850689 16947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:48:21.855417 16947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:48:21.855764 16947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:48:21.875547 16947 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:46675
I20260928 08:48:21.875566 17013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:46675 every 8 connection(s)
I20260928 08:48:21.877511 16947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:48:21.885885 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 16947
I20260928 08:48:21.885998 17014 sys_catalog.cc:263] Verifying existing consensus state
I20260928 08:48:21.886979 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:42723
--local_ip_for_outbound_sockets=127.0.97.1
--tserver_master_addrs=127.0.97.62:46675
--webserver_port=36359
--webserver_interface=127.0.97.1
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:48:21.891261 17014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e: Bootstrap starting.
I20260928 08:48:21.913838 17014 log.cc:824] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e: Log is configured to *not* fsync() on all Append() calls
I20260928 08:48:21.931646 17014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 08:48:21.932339 17014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e: Bootstrap complete.
I20260928 08:48:21.942927 17014 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c5003d43984eefb3f76ea34e47753e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46675 } }
I20260928 08:48:21.943995 17014 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8c5003d43984eefb3f76ea34e47753e, State: Initialized, Role: FOLLOWER
I20260928 08:48:21.944715 17014 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c5003d43984eefb3f76ea34e47753e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46675 } }
I20260928 08:48:21.944975 17014 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:48:21.945111 17014 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:48:21.945331 17014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [term 2 FOLLOWER]: Advancing to term 3
I20260928 08:48:21.949705 17014 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c5003d43984eefb3f76ea34e47753e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46675 } }
I20260928 08:48:21.950317 17014 leader_election.cc:302] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8c5003d43984eefb3f76ea34e47753e; no voters:
I20260928 08:48:21.951077 17014 leader_election.cc:288] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 08:48:21.951229 17018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [term 3 FOLLOWER]: Leader election won for term 3
I20260928 08:48:21.952499 17018 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [term 3 LEADER]: Becoming Leader. State: Replica: d8c5003d43984eefb3f76ea34e47753e, State: Running, Role: LEADER
I20260928 08:48:21.953187 17018 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c5003d43984eefb3f76ea34e47753e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46675 } }
I20260928 08:48:21.953569 17014 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:48:21.956166 17019 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d8c5003d43984eefb3f76ea34e47753e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c5003d43984eefb3f76ea34e47753e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46675 } } }
I20260928 08:48:21.956620 17019 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [sys.catalog]: This master's current role is: LEADER
I20260928 08:48:21.957197 17020 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8c5003d43984eefb3f76ea34e47753e. Latest consensus state: current_term: 3 leader_uuid: "d8c5003d43984eefb3f76ea34e47753e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8c5003d43984eefb3f76ea34e47753e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 46675 } } }
I20260928 08:48:21.957610 17020 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e [sys.catalog]: This master's current role is: LEADER
I20260928 08:48:21.959882 17023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:48:21.971105 17023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:48:21.973419 17023 catalog_manager.cc:1259] Loaded cluster ID: 1fe21506ceb94a12898c2c5747916494
I20260928 08:48:21.973670 17023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:48:21.979395 17023 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:48:21.981918 17023 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e: Loaded TSK: 0
I20260928 08:48:21.982260 17023 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d8c5003d43984eefb3f76ea34e47753e: Loaded TSK: 1
F20260928 08:48:21.984016 17023 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fbb393639ad google::LogMessage::Fail() at ??:0
@ 0x7fbb3936688a google::LogMessage::SendToLog() at ??:0
@ 0x7fbb393634de google::LogMessage::Flush() at ??:0
@ 0x7fbb39364a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fbb3d0c77d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fbb3d0b673e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fbb4ed921c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fbb4ed91783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fbb4ee1d2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fbb4ee1d0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fbb4ee675e9 std::function<>::operator()() at ??:0
@ 0x7fbb4ed95cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fbb4ed93c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fbb4ee1b610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fbb4ee1b439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fbb3bfaf345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fbb3bfb291c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fbb3bfb2799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fbb3bf8deaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fbb3f2676db start_thread at ??:0
@ 0x7fbb381fd61f clone at ??:0
W20260928 08:48:22.288829 17016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:48:22.289184 17016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:48:22.289357 17016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:48:22.297598 17016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:48:22.297749 17016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:48:22.297940 17016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:48:22.308107 17016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:42723
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=36359
--tserver_master_addrs=127.0.97.62:46675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:48:22.309471 17016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:48:22.310804 17016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:48:22.324347 17061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:22.324472 17062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:22.324671 17064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:48:22.325570 17016 server_base.cc:1034] running on GCE node
I20260928 08:48:22.326277 17016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:48:22.327332 17016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:48:22.328532 17016 hybrid_clock.cc:648] HybridClock initialized: now 1790585302328486 us; error 50 us; skew 500 ppm
I20260928 08:48:22.328860 17016 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:48:22 dist-test-slave-n6xs krb5kdc[15867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:48:22.332178 17016 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:48:22.333765 17016 webserver.cc:460] Webserver started at http://127.0.97.1:36359/ using document root <none> and password file <none>
I20260928 08:48:22.334489 17016 fs_manager.cc:359] Metadata directory not provided
I20260928 08:48:22.334622 17016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:48:22.340277 17016 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 08:48:22.343101 17071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:22.344590 17016 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:48:22.344763 17016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/wal
uuid: "16395cd18c07415489f77aa172a80133"
format_stamp: "Formatted at 2026-09-28 08:47:36 on dist-test-slave-n6xs"
server_key: "29ff7f520798481cab12b239ba5f69d0"
server_key_iv: "ea648a53b3fcd812f1fe8b5425aab000"
server_key_version: "encryptionkey@0"
I20260928 08:48:22.345796 17016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:48:22.364151 17016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:48:22.368666 17016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:48:22.368969 17016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:48:22.369975 17016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:48:22.371280 17016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:48:22.371416 17016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:22.371551 17016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:48:22.371639 17016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:22.406409 17016 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:42723
I20260928 08:48:22.406507 17183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:42723 every 8 connection(s)
I20260928 08:48:22.408015 17016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:48:22.413077 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:22.411044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.1:50741)
0928 08:48:22.411951 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:22.412164 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":365,"threads_started":1}
W20260928 08:48:22.413800 17184 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46675 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:46675: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
I20260928 08:48:22.414657 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:22.414313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.1:39215)
0928 08:48:22.414480 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:22.414551 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 08:48:22.415452 17186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:22.415139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.1:50001)
0928 08:48:22.415270 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:22.415346 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 08:48:22.415680 17184 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 08:48:22.416476 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 17016
I20260928 08:48:22.417057 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:45079
--local_ip_for_outbound_sockets=127.0.97.2
--tserver_master_addrs=127.0.97.62:46675
--webserver_port=40311
--webserver_interface=127.0.97.2
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:48:22.757414 17187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:48:22.757766 17187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:48:22.757952 17187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:48:22.765537 17187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:48:22.765688 17187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:48:22.765882 17187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:48:22.775542 17187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:45079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=40311
--tserver_master_addrs=127.0.97.62:46675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:48:22.776904 17187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:48:22.778225 17187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:48:22.791407 17195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:22.791457 17193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:22.791481 17192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:48:22.792358 17187 server_base.cc:1034] running on GCE node
I20260928 08:48:22.793001 17187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:48:22.793931 17187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:48:22.795167 17187 hybrid_clock.cc:648] HybridClock initialized: now 1790585302795115 us; error 45 us; skew 500 ppm
I20260928 08:48:22.795534 17187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:48:22 dist-test-slave-n6xs krb5kdc[15867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585302, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:48:22.798789 17187 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:48:22.800271 17187 webserver.cc:460] Webserver started at http://127.0.97.2:40311/ using document root <none> and password file <none>
I20260928 08:48:22.800935 17187 fs_manager.cc:359] Metadata directory not provided
I20260928 08:48:22.801090 17187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:48:22.806223 17187 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 08:48:22.809055 17202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:22.810448 17187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 08:48:22.810627 17187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/wal
uuid: "c018f42c29c74413abcfd6054fcb9be2"
format_stamp: "Formatted at 2026-09-28 08:47:36 on dist-test-slave-n6xs"
server_key: "e7acf1de1fa2ce7210205ae55746084e"
server_key_iv: "f6d85eabd0984b921b082970a1b2bf03"
server_key_version: "encryptionkey@0"
I20260928 08:48:22.811655 17187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:48:22.838809 17187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:48:22.843263 17187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:48:22.843561 17187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:48:22.844697 17187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:48:22.846055 17187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:48:22.846174 17187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:22.846320 17187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:48:22.846410 17187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:22.880841 17187 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:45079
I20260928 08:48:22.880968 17314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:45079 every 8 connection(s)
I20260928 08:48:22.882815 17187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:48:22.888142 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:22.886142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.2:33457)
0928 08:48:22.886928 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:22.887143 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":312,"threads_started":1}
W20260928 08:48:22.888863 17315 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46675 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:46675: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
I20260928 08:48:22.889722 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:22.889394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.2:46603)
0928 08:48:22.889546 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:22.889624 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 08:48:22.890514 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:22.890200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.2:35319)
0928 08:48:22.890334 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:22.890414 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 08:48:22.890755 17315 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 08:48:22.891749 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 17187
I20260928 08:48:22.892359 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:42469
--local_ip_for_outbound_sockets=127.0.97.3
--tserver_master_addrs=127.0.97.62:46675
--webserver_port=37671
--webserver_interface=127.0.97.3
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:34065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:48:23.212325 17318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:48:23.212642 17318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:48:23.212811 17318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:48:23.219986 17318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:48:23.220113 17318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:48:23.220273 17318 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:48:23.229646 17318 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:42469
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=37671
--tserver_master_addrs=127.0.97.62:46675
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:48:23.230875 17318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:48:23.232196 17318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:48:23.244746 17323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:23.244781 17326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:23.244819 17324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:48:23.246030 17318 server_base.cc:1034] running on GCE node
I20260928 08:48:23.246583 17318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:48:23.247538 17318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:48:23.248767 17318 hybrid_clock.cc:648] HybridClock initialized: now 1790585303248726 us; error 55 us; skew 500 ppm
I20260928 08:48:23.249061 17318 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:48:23 dist-test-slave-n6xs krb5kdc[15867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585303, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:48:23.252089 17318 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:48:23.253365 17318 webserver.cc:460] Webserver started at http://127.0.97.3:37671/ using document root <none> and password file <none>
I20260928 08:48:23.253935 17318 fs_manager.cc:359] Metadata directory not provided
I20260928 08:48:23.254053 17318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:48:23.259096 17318 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:48:23.261515 17333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:23.262827 17318 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 08:48:23.263018 17318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/wal
uuid: "59dc27f1dc6847b5acee11d95e81ba0f"
format_stamp: "Formatted at 2026-09-28 08:47:37 on dist-test-slave-n6xs"
server_key: "b45b76e3326428c205094c335aee1f6a"
server_key_iv: "b886b98f904466ac8534631e81540f77"
server_key_version: "encryptionkey@0"
I20260928 08:48:23.263909 17318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:48:23.275908 17318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:48:23.279973 17318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:48:23.280227 17318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:48:23.281086 17318 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:48:23.282284 17318 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:48:23.282383 17318 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:23.282529 17318 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:48:23.282614 17318 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:23.313938 17318 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42469
I20260928 08:48:23.313993 17445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42469 every 8 connection(s)
I20260928 08:48:23.315419 17318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:48:23.320245 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:23.318385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.3:55367)
0928 08:48:23.319066 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:23.319267 (+ 201us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":273,"threads_started":1}
W20260928 08:48:23.320933 17446 heartbeater.cc:644] Failed to heartbeat to 127.0.97.62:46675 (0 consecutive failures): Network error: Failed to ping master at 127.0.97.62:46675: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
I20260928 08:48:23.321794 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:23.321423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.3:51337)
0928 08:48:23.321595 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:23.321682 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 08:48:23.322592 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:23.322293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.3:36407)
0928 08:48:23.322413 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:23.322485 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 08:48:23.322889 17446 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 08:48:23.325431 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 17318
I20260928 08:48:23.417191 17450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:23.416244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.1:60023)
0928 08:48:23.416772 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:23.416889 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":249,"threads_started":1}
I20260928 08:48:23.892189 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:23.891319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.2:35263)
0928 08:48:23.891763 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:23.891911 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":216,"threads_started":1}
I20260928 08:48:24.324122 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:24.323341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.3:36963)
0928 08:48:24.323755 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:24.323890 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":198,"threads_started":1}
I20260928 08:48:24.418843 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:24.418135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.1:55445)
0928 08:48:24.418576 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:24.418678 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":236,"threads_started":1}
I20260928 08:48:24.893899 17454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:24.893101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.2:33307)
0928 08:48:24.893637 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:24.893722 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":238,"threads_started":1}
I20260928 08:48:25.325836 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:25.325050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.3:35219)
0928 08:48:25.325565 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:25.325671 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":279,"threads_started":1}
I20260928 08:48:25.420356 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:25.419618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.1:49445)
0928 08:48:25.420113 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:25.420188 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":300,"threads_started":1}
I20260928 08:48:25.895673 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:25.894868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.2:46587)
0928 08:48:25.895386 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:25.895491 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":276,"threads_started":1}
I20260928 08:48:26.327418 17458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:26.326746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.3:50881)
0928 08:48:26.327177 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:26.327250 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":233,"threads_started":1}
I20260928 08:48:26.421957 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:26.421234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.1:32851)
0928 08:48:26.421683 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:26.421760 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":223,"threads_started":1}
I20260928 08:48:26.897579 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:26.896666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.2:33553)
0928 08:48:26.897218 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:26.897313 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":245,"threads_started":1}
I20260928 08:48:27.329213 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:27.328544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.3:58847)
0928 08:48:27.328959 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:27.329028 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":208,"threads_started":1}
I20260928 08:48:27.423585 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:27.422867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.1:34725)
0928 08:48:27.423320 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:27.423399 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":237,"threads_started":1}
I20260928 08:48:27.899375 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:27.898630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.2:52197)
0928 08:48:27.899123 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:27.899204 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":252,"threads_started":1}
I20260928 08:48:28.331064 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:28.330169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.3:38225)
0928 08:48:28.330792 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:28.330865 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":396,"threads_started":1}
I20260928 08:48:28.425192 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:28.424497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.1:47237)
0928 08:48:28.424945 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:28.425034 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":233,"threads_started":1}
I20260928 08:48:28.901189 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:28.900364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.2:54891)
0928 08:48:28.900873 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:28.901026 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":286,"threads_started":1}
I20260928 08:48:29.332784 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:29.332091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.3:40589)
0928 08:48:29.332522 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:29.332621 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":217,"threads_started":1}
I20260928 08:48:29.426847 17468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:29.426043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.1:39069)
0928 08:48:29.426537 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:29.426615 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":205,"threads_started":1}
I20260928 08:48:29.902989 17469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:29.902187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.2:47539)
0928 08:48:29.902670 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:29.902754 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":214,"threads_started":1}
I20260928 08:48:30.334457 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:30.333653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.3:48521)
0928 08:48:30.334159 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:30.334239 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":253,"threads_started":1}
I20260928 08:48:30.428541 17471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:30.427716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.1:40321)
0928 08:48:30.428286 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:30.428361 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":383,"threads_started":1}
I20260928 08:48:30.904839 17472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:30.904097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.2:49527)
0928 08:48:30.904578 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:30.904668 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":244,"threads_started":1}
I20260928 08:48:31.336181 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:31.335403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.3:57617)
0928 08:48:31.335903 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:31.336020 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":229,"threads_started":1}
I20260928 08:48:31.430249 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:31.429454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.1:42175)
0928 08:48:31.429984 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:31.430073 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":260,"threads_started":1}
I20260928 08:48:31.906618 17475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:31.905822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.2:33755)
0928 08:48:31.906307 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:31.906402 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":231,"threads_started":1}
I20260928 08:48:32.337772 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:32.337055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.3:37171)
0928 08:48:32.337516 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:32.337598 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":217,"threads_started":1}
I20260928 08:48:32.432047 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:32.431201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.1:58855)
0928 08:48:32.431674 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:32.431781 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":257,"threads_started":1}
I20260928 08:48:32.908507 17478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:32.907634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.2:59039)
0928 08:48:32.908221 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:32.908309 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":379,"threads_started":1}
I20260928 08:48:33.339417 17479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:33.338709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.3:40073)
0928 08:48:33.339168 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:33.339242 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":269,"threads_started":1}
I20260928 08:48:33.433576 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:33.432885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.1:40607)
0928 08:48:33.433282 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:33.433369 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":193,"threads_started":1}
I20260928 08:48:33.910123 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:33.909406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.2:52911)
0928 08:48:33.909847 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:33.909941 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":216,"threads_started":1}
I20260928 08:48:34.341177 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:34.340499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.3:55443)
0928 08:48:34.340922 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:34.341008 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":222,"threads_started":1}
I20260928 08:48:34.435277 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:34.434576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.1:44569)
0928 08:48:34.435037 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:34.435117 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":266,"threads_started":1}
I20260928 08:48:34.911701 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:34.911060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.2:38791)
0928 08:48:34.911461 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:34.911548 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":212,"threads_started":1}
I20260928 08:48:35.342931 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:35.342168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.3:46321)
0928 08:48:35.342648 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:35.342731 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":254,"threads_started":1}
I20260928 08:48:35.437083 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:35.436216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.1:37469)
0928 08:48:35.436802 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:35.436896 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":365,"threads_started":1}
I20260928 08:48:35.913535 17487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:35.912729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.2:34965)
0928 08:48:35.913204 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:35.913294 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":240,"threads_started":1}
I20260928 08:48:36.344604 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:36.343802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.3:33861)
0928 08:48:36.344326 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:36.344411 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":208,"threads_started":1}
I20260928 08:48:36.438732 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:36.437966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.1:56845)
0928 08:48:36.438408 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:36.438500 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":230,"threads_started":1}
I20260928 08:48:36.915198 17490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:36.914491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.2:41107)
0928 08:48:36.914924 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:36.915020 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":232,"threads_started":1}
I20260928 08:48:37.346493 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:37.345648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.3:56913)
0928 08:48:37.346156 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:37.346267 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":266,"threads_started":1}
I20260928 08:48:37.440774 17492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:37.440026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.1:54595)
0928 08:48:37.440457 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:37.440535 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":216,"threads_started":1}
I20260928 08:48:37.917378 17493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:37.916527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.2:38207)
0928 08:48:37.917057 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:37.917169 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":307,"threads_started":1}
I20260928 08:48:38.332625 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 17016
I20260928 08:48:38.348503 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:38.347728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:46675 (local address 127.0.97.3:59141)
0928 08:48:38.348232 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:38.348312 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.62:46675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":285,"threads_started":1}
I20260928 08:48:38.359338 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:38.334260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52394 (local address 127.0.97.1:42723)
0928 08:48:38.335058 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:38.335079 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:38.335128 (+ 49us) server_negotiation.cc:407] Connection header received
0928 08:48:38.336848 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:38.336857 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:38.337348 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:38.337646 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:38.339097 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:38.340507 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:38.341918 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:38.342624 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:38.342773 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:38.351901 (+ 9128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:38.351921 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:38.351961 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:48:38.352005 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:38.355519 (+ 3514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:38.356354 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:38.356362 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:38.356377 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:38.356577 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:38.356976 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:38.356980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:38.356982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:38.357650 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:38.357927 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:38.358611 (+ 684us) server_negotiation.cc:299] Negotiation successful
0928 08:48:38.359034 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":374,"threads_started":1}
I20260928 08:48:38.360932 17179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:48:38.445233 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 17016
I20260928 08:48:38.462852 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 17187
I20260928 08:48:38.481209 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:38.463934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57166 (local address 127.0.97.2:45079)
0928 08:48:38.464449 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:38.464464 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:38.464501 (+ 37us) server_negotiation.cc:407] Connection header received
0928 08:48:38.466169 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:38.466183 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:38.466622 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:38.466998 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:38.468476 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:38.469743 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:38.471081 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:38.471768 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:38.472028 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:38.474628 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:38.474676 (+ 48us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:38.474700 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:48:38.474762 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:38.477695 (+ 2933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:38.478424 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:38.478440 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:38.478448 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:38.478617 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:38.479000 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:38.479006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:38.479009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:38.479556 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:38.479915 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:38.480481 (+ 566us) server_negotiation.cc:299] Negotiation successful
0928 08:48:38.480922 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":233,"threads_started":1}
I20260928 08:48:38.482604 17310 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:48:38.590564 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 17187
I20260928 08:48:38.609582 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 17318
I20260928 08:48:38.629335 17500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:38.610794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41456 (local address 127.0.97.3:42469)
0928 08:48:38.611397 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:38.611411 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:38.611455 (+ 44us) server_negotiation.cc:407] Connection header received
0928 08:48:38.613459 (+ 2004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:38.613470 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:38.613979 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:38.614315 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:38.615772 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:38.617171 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:38.618526 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:38.619287 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:38.619438 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:38.622297 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:38.622306 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:38.622324 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:48:38.622381 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:38.625536 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:38.626297 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:38.626304 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:38.626311 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:38.626501 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:38.626916 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:38.626922 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:38.626925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:38.627495 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:38.627854 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:38.628425 (+ 571us) server_negotiation.cc:299] Negotiation successful
0928 08:48:38.628992 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":298,"threads_started":1}
I20260928 08:48:38.630897 17441 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:48:38.714720 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 17318
2026-09-28T08:48:38Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64916 ms)
[----------] 30 tests from SecurityITest (296233 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:48:38 dist-test-slave-n6xs krb5kdc[17504](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:48:38 dist-test-slave-n6xs krb5kdc[17504](info): set up 2 sockets
Sep 28 08:48:38 dist-test-slave-n6xs krb5kdc[17504](info): commencing operation
krb5kdc: starting...
W20260928 08:48:39.836155 388 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:48:39 dist-test-slave-n6xs krb5kdc[17504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:48:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:48:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:48:40.095299 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43283
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:48:40.433434 17517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:48:40.433801 17517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:48:40.433967 17517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:48:40.441277 17517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:48:40.441392 17517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:48:40.441488 17517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:48:40.441583 17517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:48:40.441668 17517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:48:40.451709 17517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:48:40.453152 17517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:48:40.454471 17517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:48:40.466585 17525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:40.466543 17522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:40.466655 17523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:48:40.473572 17517 server_base.cc:1034] running on GCE node
I20260928 08:48:40.474584 17517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:48:40.476740 17517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:48:40.478046 17517 hybrid_clock.cc:648] HybridClock initialized: now 1790585320477931 us; error 111 us; skew 500 ppm
I20260928 08:48:40.478543 17517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:48:40 dist-test-slave-n6xs krb5kdc[17504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:48:40.482872 17517 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:48:40.484690 17517 webserver.cc:460] Webserver started at http://127.0.97.62:44649/ using document root <none> and password file <none>
I20260928 08:48:40.485503 17517 fs_manager.cc:359] Metadata directory not provided
I20260928 08:48:40.485646 17517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:48:40.485994 17517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:48:40.489300 17517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "c764c1ac72664222896c42535ecb249a"
format_stamp: "Formatted at 2026-09-28 08:48:40 on dist-test-slave-n6xs"
server_key: "b47e167618bd26c4b3e8b29f302a5a7b"
server_key_iv: "4ed313fc64b33323105fff33eff58e41"
server_key_version: "encryptionkey@0"
I20260928 08:48:40.490121 17517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "c764c1ac72664222896c42535ecb249a"
format_stamp: "Formatted at 2026-09-28 08:48:40 on dist-test-slave-n6xs"
server_key: "b47e167618bd26c4b3e8b29f302a5a7b"
server_key_iv: "4ed313fc64b33323105fff33eff58e41"
server_key_version: "encryptionkey@0"
I20260928 08:48:40.495616 17517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 08:48:40.499246 17532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:40.500723 17517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:48:40.500921 17517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "c764c1ac72664222896c42535ecb249a"
format_stamp: "Formatted at 2026-09-28 08:48:40 on dist-test-slave-n6xs"
server_key: "b47e167618bd26c4b3e8b29f302a5a7b"
server_key_iv: "4ed313fc64b33323105fff33eff58e41"
server_key_version: "encryptionkey@0"
I20260928 08:48:40.501125 17517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:48:40.523317 17517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:48:40.527794 17517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:48:40.528172 17517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:48:40.547726 17517 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43283
I20260928 08:48:40.547771 17583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43283 every 8 connection(s)
I20260928 08:48:40.549182 17517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:48:40.551167 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 17517
I20260928 08:48:40.551396 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:48:40.552271 388 external_mini_cluster.cc:1428] Setting key 9e543c5c32970cee99c298b51a007051
I20260928 08:48:40.554011 17584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 08:48:40 dist-test-slave-n6xs krb5kdc[17504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:48:40.568738 17584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c764c1ac72664222896c42535ecb249a: Bootstrap starting.
I20260928 08:48:40.572425 17584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c764c1ac72664222896c42535ecb249a: Neither blocks nor log segments found. Creating new log.
I20260928 08:48:40.573799 17584 log.cc:824] T 00000000000000000000000000000000 P c764c1ac72664222896c42535ecb249a: Log is configured to *not* fsync() on all Append() calls
I20260928 08:48:40.577339 17584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c764c1ac72664222896c42535ecb249a: No bootstrap required, opened a new log
I20260928 08:48:40.577477 17587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:40.554420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47680 (local address 127.0.97.62:43283)
0928 08:48:40.555278 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:40.555290 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:40.555331 (+ 41us) server_negotiation.cc:407] Connection header received
0928 08:48:40.557076 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:40.557099 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:40.557513 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:40.557868 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:40.559728 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:40.561180 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:40.563022 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:40.563694 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:40.563914 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:40.568967 (+ 5053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:40.568977 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:40.568992 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:48:40.569036 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:40.572512 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:40.573506 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:40.573513 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:40.573520 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:40.573712 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:40.574153 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:40.574156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:40.574158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:40.575024 (+ 866us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:40.575360 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:40.575949 (+ 589us) server_negotiation.cc:299] Negotiation successful
0928 08:48:40.576408 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":414,"threads_started":1}
I20260928 08:48:40.585772 17584 raft_consensus.cc:348] T 00000000000000000000000000000000 P c764c1ac72664222896c42535ecb249a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c764c1ac72664222896c42535ecb249a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43283 } }
I20260928 08:48:40.586195 17584 raft_consensus.cc:374] T 00000000000000000000000000000000 P c764c1ac72664222896c42535ecb249a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:48:40.586330 17584 raft_consensus.cc:729] T 00000000000000000000000000000000 P c764c1ac72664222896c42535ecb249a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c764c1ac72664222896c42535ecb249a, State: Initialized, Role: FOLLOWER
I20260928 08:48:40.587026 17584 consensus_queue.cc:260] T 00000000000000000000000000000000 P c764c1ac72664222896c42535ecb249a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c764c1ac72664222896c42535ecb249a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43283 } }
I20260928 08:48:40.587312 17584 raft_consensus.cc:388] T 00000000000000000000000000000000 P c764c1ac72664222896c42535ecb249a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:48:40.587471 17584 raft_consensus.cc:482] T 00000000000000000000000000000000 P c764c1ac72664222896c42535ecb249a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:48:40.587658 17584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c764c1ac72664222896c42535ecb249a [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:48:40.589850 17584 raft_consensus.cc:504] T 00000000000000000000000000000000 P c764c1ac72664222896c42535ecb249a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c764c1ac72664222896c42535ecb249a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43283 } }
I20260928 08:48:40.590368 17584 leader_election.cc:302] T 00000000000000000000000000000000 P c764c1ac72664222896c42535ecb249a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c764c1ac72664222896c42535ecb249a; no voters:
I20260928 08:48:40.591003 17584 leader_election.cc:288] T 00000000000000000000000000000000 P c764c1ac72664222896c42535ecb249a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:48:40.591174 17589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c764c1ac72664222896c42535ecb249a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:48:40.591872 17589 raft_consensus.cc:686] T 00000000000000000000000000000000 P c764c1ac72664222896c42535ecb249a [term 1 LEADER]: Becoming Leader. State: Replica: c764c1ac72664222896c42535ecb249a, State: Running, Role: LEADER
I20260928 08:48:40.592418 17589 consensus_queue.cc:237] T 00000000000000000000000000000000 P c764c1ac72664222896c42535ecb249a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c764c1ac72664222896c42535ecb249a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43283 } }
I20260928 08:48:40.593169 17584 sys_catalog.cc:564] T 00000000000000000000000000000000 P c764c1ac72664222896c42535ecb249a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:48:40.595338 17591 sys_catalog.cc:455] T 00000000000000000000000000000000 P c764c1ac72664222896c42535ecb249a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c764c1ac72664222896c42535ecb249a. Latest consensus state: current_term: 1 leader_uuid: "c764c1ac72664222896c42535ecb249a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c764c1ac72664222896c42535ecb249a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43283 } } }
I20260928 08:48:40.595753 17591 sys_catalog.cc:458] T 00000000000000000000000000000000 P c764c1ac72664222896c42535ecb249a [sys.catalog]: This master's current role is: LEADER
I20260928 08:48:40.596100 17590 sys_catalog.cc:455] T 00000000000000000000000000000000 P c764c1ac72664222896c42535ecb249a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c764c1ac72664222896c42535ecb249a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c764c1ac72664222896c42535ecb249a" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43283 } } }
I20260928 08:48:40.596376 17590 sys_catalog.cc:458] T 00000000000000000000000000000000 P c764c1ac72664222896c42535ecb249a [sys.catalog]: This master's current role is: LEADER
I20260928 08:48:40.597255 17597 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:48:40.603165 17597 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:48:40.613384 17597 catalog_manager.cc:1347] Generated new cluster ID: 87c9214b4e3b4be894f9845110bf21aa
I20260928 08:48:40.613523 17597 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:48:40.630584 17597 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:48:40.632678 17597 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:48:40.643756 17597 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c764c1ac72664222896c42535ecb249a: Generated new TSK 0
I20260928 08:48:40.644915 17597 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:48:40.720090 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43283
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:48:41.069711 17610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:48:41.070000 17610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:48:41.070182 17610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:48:41.077672 17610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:48:41.077783 17610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:48:41.077996 17610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:48:41.087924 17610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:48:41.089303 17610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:48:41.090646 17610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:48:41.104560 17615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:41.104692 17616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:41.105434 17618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:48:41.105778 17610 server_base.cc:1034] running on GCE node
I20260928 08:48:41.106432 17610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:48:41.107414 17610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:48:41.108645 17610 hybrid_clock.cc:648] HybridClock initialized: now 1790585321108577 us; error 65 us; skew 500 ppm
I20260928 08:48:41.108989 17610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:48:41 dist-test-slave-n6xs krb5kdc[17504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:48:41.112838 17610 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:48:41.114418 17610 webserver.cc:460] Webserver started at http://127.0.97.1:42757/ using document root <none> and password file <none>
I20260928 08:48:41.115259 17610 fs_manager.cc:359] Metadata directory not provided
I20260928 08:48:41.115403 17610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:48:41.115715 17610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:48:41.118880 17610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-0/data/instance:
uuid: "c2e2c865320a4c5e9236d8b1613cdebc"
format_stamp: "Formatted at 2026-09-28 08:48:41 on dist-test-slave-n6xs"
server_key: "db621c3968d15b4c0b10016bf90e330e"
server_key_iv: "c4ad5b26c33fec9baaad3470388cc913"
server_key_version: "encryptionkey@0"
I20260928 08:48:41.119644 17610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-0/wal/instance:
uuid: "c2e2c865320a4c5e9236d8b1613cdebc"
format_stamp: "Formatted at 2026-09-28 08:48:41 on dist-test-slave-n6xs"
server_key: "db621c3968d15b4c0b10016bf90e330e"
server_key_iv: "c4ad5b26c33fec9baaad3470388cc913"
server_key_version: "encryptionkey@0"
I20260928 08:48:41.125135 17610 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 08:48:41.128733 17625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:41.130169 17610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:48:41.130352 17610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-0/wal
uuid: "c2e2c865320a4c5e9236d8b1613cdebc"
format_stamp: "Formatted at 2026-09-28 08:48:41 on dist-test-slave-n6xs"
server_key: "db621c3968d15b4c0b10016bf90e330e"
server_key_iv: "c4ad5b26c33fec9baaad3470388cc913"
server_key_version: "encryptionkey@0"
I20260928 08:48:41.130563 17610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:48:41.151911 17610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:48:41.156358 17610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:48:41.156669 17610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:48:41.157691 17610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:48:41.159075 17610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:48:41.159189 17610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:41.159343 17610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:48:41.159415 17610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:41.193913 17610 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:38257
I20260928 08:48:41.193972 17737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:38257 every 8 connection(s)
I20260928 08:48:41.195344 17610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:48:41.199580 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 17610
I20260928 08:48:41.199841 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:48:41.200783 388 external_mini_cluster.cc:1428] Setting key f148361342fb7166213a2b41d3241924
Sep 28 08:48:41 dist-test-slave-n6xs krb5kdc[17504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:48:41.221071 17740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:41.198075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40843 (local address 127.0.97.62:43283)
0928 08:48:41.198607 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:41.198615 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:41.199893 (+ 1278us) server_negotiation.cc:407] Connection header received
0928 08:48:41.202412 (+ 2519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:41.202416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:41.202487 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:41.202574 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:41.204898 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:41.205742 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:41.208319 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:41.208735 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:41.208839 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:41.213486 (+ 4647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:41.213496 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:41.213504 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:48:41.213556 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:41.217323 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:41.218232 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:41.218240 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:41.218247 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:41.218414 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:41.218881 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:41.218889 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:41.218895 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:41.219282 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:41.219436 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:41.220273 (+ 837us) server_negotiation.cc:299] Negotiation successful
0928 08:48:41.220549 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":221,"threads_started":1}
I20260928 08:48:41.221695 17741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:41.198733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43283 (local address 127.0.97.1:40843)
0928 08:48:41.199684 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:41.199744 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:41.202090 (+ 2346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:41.202871 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:41.202880 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:41.203688 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:48:41.204652 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:41.204683 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:41.205918 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:41.205926 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:41.208160 (+ 2234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:41.208178 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:41.209000 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:41.209010 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:41.209377 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:41.210169 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:48:41.210213 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:48:41.213173 (+ 2960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:48:41.217520 (+ 4347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:48:41.217529 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:48:41.217544 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:48:41.218065 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:48:41.218530 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:48:41.218533 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:48:41.218535 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:48:41.218758 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:48:41.219400 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:48:41.219406 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:48:41.219786 (+ 380us) client_negotiation.cc:769] Sending connection context
0928 08:48:41.220104 (+ 318us) client_negotiation.cc:240] Negotiation successful
0928 08:48:41.220525 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":324,"threads_started":1}
I20260928 08:48:41.223363 17738 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43283
I20260928 08:48:41.223714 17738 heartbeater.cc:464] Registering TS with master...
I20260928 08:48:41.224701 17738 heartbeater.cc:511] Master 127.0.97.62:43283 requested a full tablet report, sending...
I20260928 08:48:41.227324 17549 ts_manager.cc:194] Registered new tserver with Master: c2e2c865320a4c5e9236d8b1613cdebc (127.0.97.1:38257)
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
I20260928 08:48:41.229058 17549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:40843
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 08:48:41.262754 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43283
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:48:41.605647 17744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:48:41.605952 17744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:48:41.606199 17744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:48:41.613713 17744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:48:41.613849 17744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:48:41.614051 17744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:48:41.624138 17744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:48:41.625561 17744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:48:41.626950 17744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:48:41.641100 17752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:41.641152 17749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:41.641182 17750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:48:41.642051 17744 server_base.cc:1034] running on GCE node
I20260928 08:48:41.642717 17744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:48:41.643750 17744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:48:41.645009 17744 hybrid_clock.cc:648] HybridClock initialized: now 1790585321644949 us; error 58 us; skew 500 ppm
I20260928 08:48:41.645450 17744 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:48:41 dist-test-slave-n6xs krb5kdc[17504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:48:41.648918 17744 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:48:41.650431 17744 webserver.cc:460] Webserver started at http://127.0.97.2:45997/ using document root <none> and password file <none>
I20260928 08:48:41.651191 17744 fs_manager.cc:359] Metadata directory not provided
I20260928 08:48:41.651345 17744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:48:41.651680 17744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:48:41.654845 17744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-1/data/instance:
uuid: "158749d76d0d44f98c9503149fb606e8"
format_stamp: "Formatted at 2026-09-28 08:48:41 on dist-test-slave-n6xs"
server_key: "fff20695f419715b60ef6245724c2da5"
server_key_iv: "6c4b0ea94f908b7c40f882754c877ee8"
server_key_version: "encryptionkey@0"
I20260928 08:48:41.655701 17744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-1/wal/instance:
uuid: "158749d76d0d44f98c9503149fb606e8"
format_stamp: "Formatted at 2026-09-28 08:48:41 on dist-test-slave-n6xs"
server_key: "fff20695f419715b60ef6245724c2da5"
server_key_iv: "6c4b0ea94f908b7c40f882754c877ee8"
server_key_version: "encryptionkey@0"
I20260928 08:48:41.661316 17744 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 08:48:41.665167 17759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:41.666761 17744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 08:48:41.666942 17744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-1/wal
uuid: "158749d76d0d44f98c9503149fb606e8"
format_stamp: "Formatted at 2026-09-28 08:48:41 on dist-test-slave-n6xs"
server_key: "fff20695f419715b60ef6245724c2da5"
server_key_iv: "6c4b0ea94f908b7c40f882754c877ee8"
server_key_version: "encryptionkey@0"
I20260928 08:48:41.667169 17744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:48:41.693560 17744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:48:41.698298 17744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:48:41.698603 17744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:48:41.699723 17744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:48:41.701112 17744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:48:41.701218 17744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:41.701365 17744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:48:41.701440 17744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:41.736271 17744 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:44381
I20260928 08:48:41.736320 17871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:44381 every 8 connection(s)
I20260928 08:48:41.737596 17744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:48:41.738058 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 17744
I20260928 08:48:41.738245 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:48:41.738905 388 external_mini_cluster.cc:1428] Setting key d5d82cbfde335b714ac5486f5866078f
Sep 28 08:48:41 dist-test-slave-n6xs krb5kdc[17504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585321, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:48:41.761327 17874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:41.740524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55659 (local address 127.0.97.62:43283)
0928 08:48:41.741021 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:41.741026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:41.742183 (+ 1157us) server_negotiation.cc:407] Connection header received
0928 08:48:41.744099 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:41.744107 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:41.744201 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:41.744328 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:41.745840 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:41.747050 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:41.748990 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:41.749426 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:41.749537 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:41.753920 (+ 4383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:41.753930 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:41.753938 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:48:41.753987 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:41.757768 (+ 3781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:41.758554 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:41.758564 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:41.758570 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:41.758744 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:41.759180 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:41.759187 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:41.759194 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:41.759591 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:41.759758 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:41.760531 (+ 773us) server_negotiation.cc:299] Negotiation successful
0928 08:48:41.760817 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":282,"threads_started":1}
I20260928 08:48:41.762061 17875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:41.741190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43283 (local address 127.0.97.2:55659)
0928 08:48:41.742010 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:41.742057 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:41.743766 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:41.744526 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:41.744533 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:41.745076 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:48:41.745644 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:41.745661 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:41.747255 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:41.747259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:41.748822 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:41.748839 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:41.749681 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:41.749691 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:41.750042 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:41.750864 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:48:41.750899 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:48:41.753652 (+ 2753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:48:41.757937 (+ 4285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:48:41.757944 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:48:41.757959 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:48:41.758405 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:48:41.758858 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:48:41.758861 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:48:41.758863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:48:41.759057 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:48:41.759735 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:48:41.759741 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:48:41.760147 (+ 406us) client_negotiation.cc:769] Sending connection context
0928 08:48:41.760404 (+ 257us) client_negotiation.cc:240] Negotiation successful
0928 08:48:41.760863 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":343,"threads_started":1}
I20260928 08:48:41.763739 17872 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43283
I20260928 08:48:41.764134 17872 heartbeater.cc:464] Registering TS with master...
I20260928 08:48:41.764895 17872 heartbeater.cc:511] Master 127.0.97.62:43283 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260928 08:48:41.766513 17549 ts_manager.cc:194] Registered new tserver with Master: 158749d76d0d44f98c9503149fb606e8 (127.0.97.2:44381)
I20260928 08:48:41.767532 17549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:55659
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:48:41.800608 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43283
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:42727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:48:42.191135 17878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:48:42.191501 17878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:48:42.191674 17878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:48:42.199988 17878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:48:42.200147 17878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:48:42.200348 17878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:48:42.210724 17878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:42727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:43283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:48:42.212240 17878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:48:42.213593 17878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:48:42.230548 17884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:42.230564 17883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:48:42.231510 17878 server_base.cc:1034] running on GCE node
W20260928 08:48:42.232124 17886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:48:42.232434 17738 heartbeater.cc:503] Master 127.0.97.62:43283 was elected leader, sending a full tablet report...
I20260928 08:48:42.233431 17878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:48:42.234647 17878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:48:42.235920 17878 hybrid_clock.cc:648] HybridClock initialized: now 1790585322235842 us; error 66 us; skew 500 ppm
I20260928 08:48:42.236428 17878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:48:42 dist-test-slave-n6xs krb5kdc[17504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:48:42.240711 17878 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:48:42.242497 17878 webserver.cc:460] Webserver started at http://127.0.97.3:44767/ using document root <none> and password file <none>
I20260928 08:48:42.243228 17878 fs_manager.cc:359] Metadata directory not provided
I20260928 08:48:42.243379 17878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:48:42.243697 17878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:48:42.246898 17878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-2/data/instance:
uuid: "8c27228bc47d4f4e80efdea7487a96d0"
format_stamp: "Formatted at 2026-09-28 08:48:42 on dist-test-slave-n6xs"
server_key: "804117e394148d76fba65562a4daf472"
server_key_iv: "37d23c8f23769163761091c9b8a07d3b"
server_key_version: "encryptionkey@0"
I20260928 08:48:42.247714 17878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-2/wal/instance:
uuid: "8c27228bc47d4f4e80efdea7487a96d0"
format_stamp: "Formatted at 2026-09-28 08:48:42 on dist-test-slave-n6xs"
server_key: "804117e394148d76fba65562a4daf472"
server_key_iv: "37d23c8f23769163761091c9b8a07d3b"
server_key_version: "encryptionkey@0"
I20260928 08:48:42.253545 17878 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 08:48:42.257591 17893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:42.259203 17878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:48:42.259400 17878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-2/wal
uuid: "8c27228bc47d4f4e80efdea7487a96d0"
format_stamp: "Formatted at 2026-09-28 08:48:42 on dist-test-slave-n6xs"
server_key: "804117e394148d76fba65562a4daf472"
server_key_iv: "37d23c8f23769163761091c9b8a07d3b"
server_key_version: "encryptionkey@0"
I20260928 08:48:42.259670 17878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:48:42.286422 17878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:48:42.291364 17878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:48:42.291692 17878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:48:42.292897 17878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:48:42.294612 17878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:48:42.294713 17878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:42.294858 17878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:48:42.294929 17878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:42.335911 17878 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:39315
I20260928 08:48:42.335922 18005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:39315 every 8 connection(s)
I20260928 08:48:42.337319 17878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:48:42.338187 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 17878
I20260928 08:48:42.338371 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:48:42.338917 388 external_mini_cluster.cc:1428] Setting key aa6b3dc9be3ea75cd18c7f488ef0de58
Sep 28 08:48:42 dist-test-slave-n6xs krb5kdc[17504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:48:42.360262 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:42.340247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:50533 (local address 127.0.97.62:43283)
0928 08:48:42.340909 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:42.340935 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:42.342222 (+ 1287us) server_negotiation.cc:407] Connection header received
0928 08:48:42.344338 (+ 2116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:42.344344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:42.344430 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:42.344565 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:42.346174 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:42.347097 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:42.349088 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:42.349514 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:42.349621 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:42.353584 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:42.353589 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:42.353593 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:48:42.353630 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:42.356893 (+ 3263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:42.357800 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:42.357805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:42.357807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:42.357942 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:42.358386 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:42.358390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:42.358393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:42.358745 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:42.358870 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:42.359678 (+ 808us) server_negotiation.cc:299] Negotiation successful
0928 08:48:42.359972 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":425,"threads_started":1}
I20260928 08:48:42.361306 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:42.341065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:43283 (local address 127.0.97.3:50533)
0928 08:48:42.342039 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:42.342096 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:42.344061 (+ 1965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:42.344819 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:42.344828 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:42.345389 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:48:42.345997 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:42.346015 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:42.347286 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:42.347295 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:42.348925 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:42.348938 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:42.349726 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:42.349731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:42.349937 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:42.350721 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:48:42.350746 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:48:42.353250 (+ 2504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:48:42.357096 (+ 3846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:48:42.357105 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:48:42.357140 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:48:42.357653 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:48:42.358044 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:48:42.358047 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:48:42.358049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:48:42.358261 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:48:42.358877 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:48:42.358884 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:48:42.359270 (+ 386us) client_negotiation.cc:769] Sending connection context
0928 08:48:42.359516 (+ 246us) client_negotiation.cc:240] Negotiation successful
0928 08:48:42.360083 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":393,"threads_started":1}
I20260928 08:48:42.362943 18006 heartbeater.cc:347] Connected to a master server at 127.0.97.62:43283
I20260928 08:48:42.363286 18006 heartbeater.cc:464] Registering TS with master...
I20260928 08:48:42.364190 18006 heartbeater.cc:511] Master 127.0.97.62:43283 requested a full tablet report, sending...
I20260928 08:48:42.365877 17549 ts_manager.cc:194] Registered new tserver with Master: 8c27228bc47d4f4e80efdea7487a96d0 (127.0.97.3:39315)
I20260928 08:48:42.367120 17549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:50533
I20260928 08:48:42.368196 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:48:42.388383 18008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:42.373685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47694 (local address 127.0.97.62:43283)
0928 08:48:42.373888 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:42.373892 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:42.374389 (+ 497us) server_negotiation.cc:407] Connection header received
0928 08:48:42.374617 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:42.374622 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:42.374728 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:42.374829 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:42.376654 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:42.377408 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:42.378958 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:42.379389 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:42.379494 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:42.382445 (+ 2951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:42.382450 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:42.382453 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:48:42.382492 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:42.385273 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:42.386101 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:42.386105 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:42.386107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:42.386257 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:42.386666 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:42.386669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:42.386671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:42.387009 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:42.387207 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:42.387664 (+ 457us) server_negotiation.cc:299] Negotiation successful
0928 08:48:42.387977 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 08:48:42.393018 17548 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47694:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:48:42.400653 17548 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:48:42.429141 18020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:42.414099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51170 (local address 127.0.97.2:44381)
0928 08:48:42.416291 (+ 2192us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:42.416301 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:42.416334 (+ 33us) server_negotiation.cc:407] Connection header received
0928 08:48:42.416448 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:42.416459 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:42.416768 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:42.416988 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:48:42.418273 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:42.420412 (+ 2139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:42.424545 (+ 4133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:42.427149 (+ 2604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:42.427387 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:42.427689 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:42.428621 (+ 932us) server_negotiation.cc:299] Negotiation successful
0928 08:48:42.428813 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2014,"thread_start_us":1881,"threads_started":1}
I20260928 08:48:42.429196 18019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:42.413899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44381 (local address 127.0.0.1:51170)
0928 08:48:42.415258 (+ 1359us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:42.415307 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:42.415532 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:42.417297 (+ 1765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:42.417310 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:42.417367 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:48:42.418003 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:42.418032 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:42.420672 (+ 2640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:42.420682 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:42.424187 (+ 3505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:42.424222 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:42.427915 (+ 3693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:42.427926 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:42.428261 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:42.428334 (+ 73us) client_negotiation.cc:769] Sending connection context
0928 08:48:42.428517 (+ 183us) client_negotiation.cc:240] Negotiation successful
0928 08:48:42.428768 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1036,"thread_start_us":481,"threads_started":1}
I20260928 08:48:42.433866 17807 tablet_service.cc:1467] Processing CreateTablet for tablet f1310b3cca2048b3b02f4dc0c63adf1f (DEFAULT_TABLE table=test-table [id=c5d2b2aca1d44fd6a0d50b400953df73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:48:42.435576 17807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1310b3cca2048b3b02f4dc0c63adf1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:48:42.437691 18023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:42.416170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38257 (local address 127.0.0.1:48812)
0928 08:48:42.428456 (+ 12286us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:42.428485 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:42.429761 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:42.430374 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:42.430378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:42.430406 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:48:42.430775 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:42.430787 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:42.432509 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:42.432515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:42.434740 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:42.434756 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:42.436791 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:42.436796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:42.436920 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:42.436945 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 08:48:42.437078 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 08:48:42.437228 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12120,"mutex_wait_us":54,"thread_start_us":12472,"threads_started":1}
I20260928 08:48:42.438485 18022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:42.416624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48812 (local address 127.0.97.1:38257)
0928 08:48:42.428466 (+ 11842us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:42.428478 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:42.428606 (+ 128us) server_negotiation.cc:407] Connection header received
0928 08:48:42.429982 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:42.429988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:42.430177 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:42.430392 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:48:42.431010 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:42.432290 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:42.434918 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:42.436337 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:42.436475 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:42.436673 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:42.437955 (+ 1282us) server_negotiation.cc:299] Negotiation successful
0928 08:48:42.438166 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11640,"thread_start_us":11494,"threads_started":1}
I20260928 08:48:42.438791 18021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:42.416700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:39315 (local address 127.0.0.1:57118)
0928 08:48:42.428869 (+ 12169us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:42.428886 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:42.429037 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:42.430803 (+ 1766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:42.430810 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:42.430843 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:48:42.431238 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:42.431252 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:42.433297 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:42.433302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:42.435656 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:42.435672 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:42.437895 (+ 2223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:42.437901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:42.438102 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:42.438148 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 08:48:42.438249 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 08:48:42.438390 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12045,"mutex_wait_us":4,"thread_start_us":10889,"threads_started":1}
I20260928 08:48:42.439739 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:42.417067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57118 (local address 127.0.97.3:39315)
0928 08:48:42.426430 (+ 9363us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:42.426444 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:42.429193 (+ 2749us) server_negotiation.cc:407] Connection header received
0928 08:48:42.429311 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:42.429319 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:42.429660 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:42.430808 (+ 1148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:48:42.431443 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:42.433081 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:42.435889 (+ 2808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:42.437328 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:42.437475 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:42.437711 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:42.439170 (+ 1459us) server_negotiation.cc:299] Negotiation successful
0928 08:48:42.439375 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9078,"thread_start_us":8750,"threads_started":1}
I20260928 08:48:42.442690 17673 tablet_service.cc:1467] Processing CreateTablet for tablet f1310b3cca2048b3b02f4dc0c63adf1f (DEFAULT_TABLE table=test-table [id=c5d2b2aca1d44fd6a0d50b400953df73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:48:42.443485 17941 tablet_service.cc:1467] Processing CreateTablet for tablet f1310b3cca2048b3b02f4dc0c63adf1f (DEFAULT_TABLE table=test-table [id=c5d2b2aca1d44fd6a0d50b400953df73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:48:42.444473 17673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1310b3cca2048b3b02f4dc0c63adf1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:48:42.444963 17941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1310b3cca2048b3b02f4dc0c63adf1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:48:42.451233 18025 tablet_bootstrap.cc:492] T f1310b3cca2048b3b02f4dc0c63adf1f P 158749d76d0d44f98c9503149fb606e8: Bootstrap starting.
I20260928 08:48:42.454595 18025 tablet_bootstrap.cc:654] T f1310b3cca2048b3b02f4dc0c63adf1f P 158749d76d0d44f98c9503149fb606e8: Neither blocks nor log segments found. Creating new log.
I20260928 08:48:42.455792 18025 log.cc:824] T f1310b3cca2048b3b02f4dc0c63adf1f P 158749d76d0d44f98c9503149fb606e8: Log is configured to *not* fsync() on all Append() calls
I20260928 08:48:42.460605 18027 tablet_bootstrap.cc:492] T f1310b3cca2048b3b02f4dc0c63adf1f P 8c27228bc47d4f4e80efdea7487a96d0: Bootstrap starting.
I20260928 08:48:42.461342 18025 tablet_bootstrap.cc:492] T f1310b3cca2048b3b02f4dc0c63adf1f P 158749d76d0d44f98c9503149fb606e8: No bootstrap required, opened a new log
I20260928 08:48:42.461560 18028 tablet_bootstrap.cc:492] T f1310b3cca2048b3b02f4dc0c63adf1f P c2e2c865320a4c5e9236d8b1613cdebc: Bootstrap starting.
I20260928 08:48:42.461722 18025 ts_tablet_manager.cc:1397] T f1310b3cca2048b3b02f4dc0c63adf1f P 158749d76d0d44f98c9503149fb606e8: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260928 08:48:42.465852 18027 tablet_bootstrap.cc:654] T f1310b3cca2048b3b02f4dc0c63adf1f P 8c27228bc47d4f4e80efdea7487a96d0: Neither blocks nor log segments found. Creating new log.
I20260928 08:48:42.467097 18028 tablet_bootstrap.cc:654] T f1310b3cca2048b3b02f4dc0c63adf1f P c2e2c865320a4c5e9236d8b1613cdebc: Neither blocks nor log segments found. Creating new log.
I20260928 08:48:42.467450 18027 log.cc:824] T f1310b3cca2048b3b02f4dc0c63adf1f P 8c27228bc47d4f4e80efdea7487a96d0: Log is configured to *not* fsync() on all Append() calls
I20260928 08:48:42.468575 18028 log.cc:824] T f1310b3cca2048b3b02f4dc0c63adf1f P c2e2c865320a4c5e9236d8b1613cdebc: Log is configured to *not* fsync() on all Append() calls
I20260928 08:48:42.471124 18027 tablet_bootstrap.cc:492] T f1310b3cca2048b3b02f4dc0c63adf1f P 8c27228bc47d4f4e80efdea7487a96d0: No bootstrap required, opened a new log
I20260928 08:48:42.471429 18027 ts_tablet_manager.cc:1397] T f1310b3cca2048b3b02f4dc0c63adf1f P 8c27228bc47d4f4e80efdea7487a96d0: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260928 08:48:42.471551 18028 tablet_bootstrap.cc:492] T f1310b3cca2048b3b02f4dc0c63adf1f P c2e2c865320a4c5e9236d8b1613cdebc: No bootstrap required, opened a new log
I20260928 08:48:42.471884 18028 ts_tablet_manager.cc:1397] T f1310b3cca2048b3b02f4dc0c63adf1f P c2e2c865320a4c5e9236d8b1613cdebc: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20260928 08:48:42.472648 18025 raft_consensus.cc:348] T f1310b3cca2048b3b02f4dc0c63adf1f P 158749d76d0d44f98c9503149fb606e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158749d76d0d44f98c9503149fb606e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44381 } } peers { permanent_uuid: "c2e2c865320a4c5e9236d8b1613cdebc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38257 } } peers { permanent_uuid: "8c27228bc47d4f4e80efdea7487a96d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39315 } }
I20260928 08:48:42.473160 18025 raft_consensus.cc:374] T f1310b3cca2048b3b02f4dc0c63adf1f P 158749d76d0d44f98c9503149fb606e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:48:42.473311 18025 raft_consensus.cc:729] T f1310b3cca2048b3b02f4dc0c63adf1f P 158749d76d0d44f98c9503149fb606e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158749d76d0d44f98c9503149fb606e8, State: Initialized, Role: FOLLOWER
I20260928 08:48:42.474016 18025 consensus_queue.cc:260] T f1310b3cca2048b3b02f4dc0c63adf1f P 158749d76d0d44f98c9503149fb606e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158749d76d0d44f98c9503149fb606e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44381 } } peers { permanent_uuid: "c2e2c865320a4c5e9236d8b1613cdebc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38257 } } peers { permanent_uuid: "8c27228bc47d4f4e80efdea7487a96d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39315 } }
I20260928 08:48:42.475257 17872 heartbeater.cc:503] Master 127.0.97.62:43283 was elected leader, sending a full tablet report...
I20260928 08:48:42.476030 18025 ts_tablet_manager.cc:1428] T f1310b3cca2048b3b02f4dc0c63adf1f P 158749d76d0d44f98c9503149fb606e8: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260928 08:48:42.481751 18027 raft_consensus.cc:348] T f1310b3cca2048b3b02f4dc0c63adf1f P 8c27228bc47d4f4e80efdea7487a96d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158749d76d0d44f98c9503149fb606e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44381 } } peers { permanent_uuid: "c2e2c865320a4c5e9236d8b1613cdebc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38257 } } peers { permanent_uuid: "8c27228bc47d4f4e80efdea7487a96d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39315 } }
I20260928 08:48:42.481935 18028 raft_consensus.cc:348] T f1310b3cca2048b3b02f4dc0c63adf1f P c2e2c865320a4c5e9236d8b1613cdebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158749d76d0d44f98c9503149fb606e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44381 } } peers { permanent_uuid: "c2e2c865320a4c5e9236d8b1613cdebc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38257 } } peers { permanent_uuid: "8c27228bc47d4f4e80efdea7487a96d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39315 } }
I20260928 08:48:42.482275 18027 raft_consensus.cc:374] T f1310b3cca2048b3b02f4dc0c63adf1f P 8c27228bc47d4f4e80efdea7487a96d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:48:42.482374 18027 raft_consensus.cc:729] T f1310b3cca2048b3b02f4dc0c63adf1f P 8c27228bc47d4f4e80efdea7487a96d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c27228bc47d4f4e80efdea7487a96d0, State: Initialized, Role: FOLLOWER
I20260928 08:48:42.482414 18028 raft_consensus.cc:374] T f1310b3cca2048b3b02f4dc0c63adf1f P c2e2c865320a4c5e9236d8b1613cdebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:48:42.482524 18028 raft_consensus.cc:729] T f1310b3cca2048b3b02f4dc0c63adf1f P c2e2c865320a4c5e9236d8b1613cdebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2e2c865320a4c5e9236d8b1613cdebc, State: Initialized, Role: FOLLOWER
I20260928 08:48:42.483144 18027 consensus_queue.cc:260] T f1310b3cca2048b3b02f4dc0c63adf1f P 8c27228bc47d4f4e80efdea7487a96d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158749d76d0d44f98c9503149fb606e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44381 } } peers { permanent_uuid: "c2e2c865320a4c5e9236d8b1613cdebc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38257 } } peers { permanent_uuid: "8c27228bc47d4f4e80efdea7487a96d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39315 } }
I20260928 08:48:42.483223 18028 consensus_queue.cc:260] T f1310b3cca2048b3b02f4dc0c63adf1f P c2e2c865320a4c5e9236d8b1613cdebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158749d76d0d44f98c9503149fb606e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44381 } } peers { permanent_uuid: "c2e2c865320a4c5e9236d8b1613cdebc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38257 } } peers { permanent_uuid: "8c27228bc47d4f4e80efdea7487a96d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39315 } }
I20260928 08:48:42.485210 18006 heartbeater.cc:503] Master 127.0.97.62:43283 was elected leader, sending a full tablet report...
I20260928 08:48:42.485715 18028 ts_tablet_manager.cc:1428] T f1310b3cca2048b3b02f4dc0c63adf1f P c2e2c865320a4c5e9236d8b1613cdebc: Time spent starting tablet: real 0.014s user 0.005s sys 0.008s
I20260928 08:48:42.485867 18027 ts_tablet_manager.cc:1428] T f1310b3cca2048b3b02f4dc0c63adf1f P 8c27228bc47d4f4e80efdea7487a96d0: Time spent starting tablet: real 0.014s user 0.012s sys 0.002s
W20260928 08:48:42.489822 17873 tablet.cc:2369] T f1310b3cca2048b3b02f4dc0c63adf1f P 158749d76d0d44f98c9503149fb606e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:48:42.589092 18007 tablet.cc:2369] T f1310b3cca2048b3b02f4dc0c63adf1f P 8c27228bc47d4f4e80efdea7487a96d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:48:42.656770 18032 raft_consensus.cc:482] T f1310b3cca2048b3b02f4dc0c63adf1f P 8c27228bc47d4f4e80efdea7487a96d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:48:42.657109 18032 raft_consensus.cc:504] T f1310b3cca2048b3b02f4dc0c63adf1f P 8c27228bc47d4f4e80efdea7487a96d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158749d76d0d44f98c9503149fb606e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44381 } } peers { permanent_uuid: "c2e2c865320a4c5e9236d8b1613cdebc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38257 } } peers { permanent_uuid: "8c27228bc47d4f4e80efdea7487a96d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39315 } }
I20260928 08:48:42.658941 18032 leader_election.cc:288] T f1310b3cca2048b3b02f4dc0c63adf1f P 8c27228bc47d4f4e80efdea7487a96d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 158749d76d0d44f98c9503149fb606e8 (127.0.97.2:44381), c2e2c865320a4c5e9236d8b1613cdebc (127.0.97.1:38257)
I20260928 08:48:42.665912 18009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:42.659146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:44381 (local address 127.0.97.3:36895)
0928 08:48:42.659387 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:42.659410 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:42.659540 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:42.660130 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:42.660142 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:42.660174 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:48:42.660393 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:42.660403 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:42.661647 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:42.661651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:42.663597 (+ 1946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:42.663612 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:42.665118 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:42.665133 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:42.665330 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:42.665368 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 08:48:42.665467 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 08:48:42.665606 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 08:48:42.666105 18020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:42.659466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36895 (local address 127.0.97.2:44381)
0928 08:48:42.659690 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:42.659707 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:42.659731 (+ 24us) server_negotiation.cc:407] Connection header received
0928 08:48:42.659863 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:42.659871 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:42.659965 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:42.660089 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:48:42.660545 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:42.661496 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:42.663760 (+ 2264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:42.664902 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:42.665077 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:42.665173 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:42.665665 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 08:48:42.665798 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 08:48:42.666744 18022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:42.659552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48541 (local address 127.0.97.1:38257)
0928 08:48:42.659851 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:42.659859 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:42.660426 (+ 567us) server_negotiation.cc:407] Connection header received
0928 08:48:42.660555 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:42.660560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:42.660640 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:42.660734 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:48:42.661193 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:42.662031 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:42.664298 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:42.665568 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:42.665703 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:42.665804 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:42.666340 (+ 536us) server_negotiation.cc:299] Negotiation successful
0928 08:48:42.666488 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260928 08:48:42.667265 18034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:42.659460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:38257 (local address 127.0.97.3:48541)
0928 08:48:42.660303 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:42.660328 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:42.660450 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:42.660754 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:42.660756 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:42.660774 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:48:42.661046 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:42.661052 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:42.662191 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:42.662196 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:42.664145 (+ 1949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:42.664156 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:42.665945 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:42.665952 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:42.666129 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:42.666164 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 08:48:42.666930 (+ 766us) client_negotiation.cc:240] Negotiation successful
0928 08:48:42.667043 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":482,"threads_started":1}
I20260928 08:48:42.667162 17827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1310b3cca2048b3b02f4dc0c63adf1f" candidate_uuid: "8c27228bc47d4f4e80efdea7487a96d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "158749d76d0d44f98c9503149fb606e8" is_pre_election: true
I20260928 08:48:42.667894 17827 raft_consensus.cc:2454] T f1310b3cca2048b3b02f4dc0c63adf1f P 158749d76d0d44f98c9503149fb606e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c27228bc47d4f4e80efdea7487a96d0 in term 0.
I20260928 08:48:42.668274 17693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1310b3cca2048b3b02f4dc0c63adf1f" candidate_uuid: "8c27228bc47d4f4e80efdea7487a96d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2e2c865320a4c5e9236d8b1613cdebc" is_pre_election: true
I20260928 08:48:42.668635 17896 leader_election.cc:302] T f1310b3cca2048b3b02f4dc0c63adf1f P 8c27228bc47d4f4e80efdea7487a96d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158749d76d0d44f98c9503149fb606e8, 8c27228bc47d4f4e80efdea7487a96d0; no voters:
I20260928 08:48:42.669003 17693 raft_consensus.cc:2454] T f1310b3cca2048b3b02f4dc0c63adf1f P c2e2c865320a4c5e9236d8b1613cdebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c27228bc47d4f4e80efdea7487a96d0 in term 0.
I20260928 08:48:42.669057 18032 raft_consensus.cc:2781] T f1310b3cca2048b3b02f4dc0c63adf1f P 8c27228bc47d4f4e80efdea7487a96d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:48:42.669211 18032 raft_consensus.cc:482] T f1310b3cca2048b3b02f4dc0c63adf1f P 8c27228bc47d4f4e80efdea7487a96d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:48:42.669314 18032 raft_consensus.cc:3037] T f1310b3cca2048b3b02f4dc0c63adf1f P 8c27228bc47d4f4e80efdea7487a96d0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:48:42.672139 18032 raft_consensus.cc:504] T f1310b3cca2048b3b02f4dc0c63adf1f P 8c27228bc47d4f4e80efdea7487a96d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158749d76d0d44f98c9503149fb606e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44381 } } peers { permanent_uuid: "c2e2c865320a4c5e9236d8b1613cdebc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38257 } } peers { permanent_uuid: "8c27228bc47d4f4e80efdea7487a96d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39315 } }
I20260928 08:48:42.672900 18032 leader_election.cc:288] T f1310b3cca2048b3b02f4dc0c63adf1f P 8c27228bc47d4f4e80efdea7487a96d0 [CANDIDATE]: Term 1 election: Requested vote from peers 158749d76d0d44f98c9503149fb606e8 (127.0.97.2:44381), c2e2c865320a4c5e9236d8b1613cdebc (127.0.97.1:38257)
I20260928 08:48:42.673331 17827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1310b3cca2048b3b02f4dc0c63adf1f" candidate_uuid: "8c27228bc47d4f4e80efdea7487a96d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "158749d76d0d44f98c9503149fb606e8"
I20260928 08:48:42.673468 17693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1310b3cca2048b3b02f4dc0c63adf1f" candidate_uuid: "8c27228bc47d4f4e80efdea7487a96d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2e2c865320a4c5e9236d8b1613cdebc"
I20260928 08:48:42.673564 17827 raft_consensus.cc:3037] T f1310b3cca2048b3b02f4dc0c63adf1f P 158749d76d0d44f98c9503149fb606e8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:48:42.673686 17693 raft_consensus.cc:3037] T f1310b3cca2048b3b02f4dc0c63adf1f P c2e2c865320a4c5e9236d8b1613cdebc [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:48:42.676911 17827 raft_consensus.cc:2454] T f1310b3cca2048b3b02f4dc0c63adf1f P 158749d76d0d44f98c9503149fb606e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c27228bc47d4f4e80efdea7487a96d0 in term 1.
I20260928 08:48:42.676911 17693 raft_consensus.cc:2454] T f1310b3cca2048b3b02f4dc0c63adf1f P c2e2c865320a4c5e9236d8b1613cdebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c27228bc47d4f4e80efdea7487a96d0 in term 1.
I20260928 08:48:42.677381 17896 leader_election.cc:302] T f1310b3cca2048b3b02f4dc0c63adf1f P 8c27228bc47d4f4e80efdea7487a96d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 158749d76d0d44f98c9503149fb606e8, 8c27228bc47d4f4e80efdea7487a96d0; no voters:
I20260928 08:48:42.677716 18032 raft_consensus.cc:2781] T f1310b3cca2048b3b02f4dc0c63adf1f P 8c27228bc47d4f4e80efdea7487a96d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:48:42.678196 18032 raft_consensus.cc:686] T f1310b3cca2048b3b02f4dc0c63adf1f P 8c27228bc47d4f4e80efdea7487a96d0 [term 1 LEADER]: Becoming Leader. State: Replica: 8c27228bc47d4f4e80efdea7487a96d0, State: Running, Role: LEADER
I20260928 08:48:42.678694 18032 consensus_queue.cc:237] T f1310b3cca2048b3b02f4dc0c63adf1f P 8c27228bc47d4f4e80efdea7487a96d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158749d76d0d44f98c9503149fb606e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44381 } } peers { permanent_uuid: "c2e2c865320a4c5e9236d8b1613cdebc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38257 } } peers { permanent_uuid: "8c27228bc47d4f4e80efdea7487a96d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39315 } }
I20260928 08:48:42.683305 17548 catalog_manager.cc:5495] T f1310b3cca2048b3b02f4dc0c63adf1f P 8c27228bc47d4f4e80efdea7487a96d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c27228bc47d4f4e80efdea7487a96d0 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "8c27228bc47d4f4e80efdea7487a96d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "158749d76d0d44f98c9503149fb606e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44381 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2e2c865320a4c5e9236d8b1613cdebc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c27228bc47d4f4e80efdea7487a96d0" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 39315 } health_report { overall_health: HEALTHY } } }
W20260928 08:48:42.698305 17739 tablet.cc:2369] T f1310b3cca2048b3b02f4dc0c63adf1f P c2e2c865320a4c5e9236d8b1613cdebc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:48:42.736681 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:42.730892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57134 (local address 127.0.97.3:39315)
0928 08:48:42.731182 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:42.731186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:42.731205 (+ 19us) server_negotiation.cc:407] Connection header received
0928 08:48:42.731418 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:42.731422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:42.731502 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:42.731669 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:48:42.732071 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:42.732809 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:42.734220 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:42.734722 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:42.734814 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:42.735297 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:48:42.736066 (+ 769us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:48:42.736244 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:42.736337 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 08:48:42.736407 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260928 08:48:42.745304 17827 raft_consensus.cc:1261] T f1310b3cca2048b3b02f4dc0c63adf1f P 158749d76d0d44f98c9503149fb606e8 [term 1 FOLLOWER]: Refusing update from remote peer 8c27228bc47d4f4e80efdea7487a96d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:48:42.745422 17693 raft_consensus.cc:1261] T f1310b3cca2048b3b02f4dc0c63adf1f P c2e2c865320a4c5e9236d8b1613cdebc [term 1 FOLLOWER]: Refusing update from remote peer 8c27228bc47d4f4e80efdea7487a96d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:48:42.746407 18035 consensus_queue.cc:1035] T f1310b3cca2048b3b02f4dc0c63adf1f P 8c27228bc47d4f4e80efdea7487a96d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2e2c865320a4c5e9236d8b1613cdebc" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 38257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:48:42.746805 18032 consensus_queue.cc:1035] T f1310b3cca2048b3b02f4dc0c63adf1f P 8c27228bc47d4f4e80efdea7487a96d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "158749d76d0d44f98c9503149fb606e8" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 44381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:48:42.776167 17548 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:48:42.776465 17548 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:48:42.780122 17548 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c764c1ac72664222896c42535ecb249a: Sending DeleteTablet for 3 replicas of tablet f1310b3cca2048b3b02f4dc0c63adf1f
I20260928 08:48:42.781468 17673 tablet_service.cc:1514] Processing DeleteTablet for tablet f1310b3cca2048b3b02f4dc0c63adf1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:48:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:48812
I20260928 08:48:42.781482 17941 tablet_service.cc:1514] Processing DeleteTablet for tablet f1310b3cca2048b3b02f4dc0c63adf1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:48:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:57118
I20260928 08:48:42.781471 17807 tablet_service.cc:1514] Processing DeleteTablet for tablet f1310b3cca2048b3b02f4dc0c63adf1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:48:42 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:51170
I20260928 08:48:42.789916 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 17610
I20260928 08:48:42.795275 18048 tablet_replica.cc:330] T f1310b3cca2048b3b02f4dc0c63adf1f P 8c27228bc47d4f4e80efdea7487a96d0: stopping tablet replica
I20260928 08:48:42.795945 18048 raft_consensus.cc:2229] T f1310b3cca2048b3b02f4dc0c63adf1f P 8c27228bc47d4f4e80efdea7487a96d0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:48:42.796753 18050 tablet_replica.cc:330] T f1310b3cca2048b3b02f4dc0c63adf1f P 158749d76d0d44f98c9503149fb606e8: stopping tablet replica
I20260928 08:48:42.796955 18048 raft_consensus.cc:2258] T f1310b3cca2048b3b02f4dc0c63adf1f P 8c27228bc47d4f4e80efdea7487a96d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:48:42.797514 18050 raft_consensus.cc:2229] T f1310b3cca2048b3b02f4dc0c63adf1f P 158749d76d0d44f98c9503149fb606e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:48:42.798185 18050 raft_consensus.cc:2258] T f1310b3cca2048b3b02f4dc0c63adf1f P 158749d76d0d44f98c9503149fb606e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:48:42.799219 18049 tablet_replica.cc:330] T f1310b3cca2048b3b02f4dc0c63adf1f P c2e2c865320a4c5e9236d8b1613cdebc: stopping tablet replica
I20260928 08:48:42.799605 18049 raft_consensus.cc:2229] T f1310b3cca2048b3b02f4dc0c63adf1f P c2e2c865320a4c5e9236d8b1613cdebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:48:42.800101 18049 raft_consensus.cc:2258] T f1310b3cca2048b3b02f4dc0c63adf1f P c2e2c865320a4c5e9236d8b1613cdebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:48:42.801080 18050 ts_tablet_manager.cc:1905] T f1310b3cca2048b3b02f4dc0c63adf1f P 158749d76d0d44f98c9503149fb606e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:48:42.801126 18048 ts_tablet_manager.cc:1905] T f1310b3cca2048b3b02f4dc0c63adf1f P 8c27228bc47d4f4e80efdea7487a96d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:48:42.801749 18049 ts_tablet_manager.cc:1905] T f1310b3cca2048b3b02f4dc0c63adf1f P c2e2c865320a4c5e9236d8b1613cdebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:48:42.808352 18049 ts_tablet_manager.cc:1918] T f1310b3cca2048b3b02f4dc0c63adf1f P c2e2c865320a4c5e9236d8b1613cdebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:48:42.808575 18049 log.cc:1196] T f1310b3cca2048b3b02f4dc0c63adf1f P c2e2c865320a4c5e9236d8b1613cdebc: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-0/wal/wals/f1310b3cca2048b3b02f4dc0c63adf1f
I20260928 08:48:42.809238 18049 ts_tablet_manager.cc:1939] T f1310b3cca2048b3b02f4dc0c63adf1f P c2e2c865320a4c5e9236d8b1613cdebc: Deleting consensus metadata
I20260928 08:48:42.809286 18050 ts_tablet_manager.cc:1918] T f1310b3cca2048b3b02f4dc0c63adf1f P 158749d76d0d44f98c9503149fb606e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:48:42.809453 18050 log.cc:1196] T f1310b3cca2048b3b02f4dc0c63adf1f P 158749d76d0d44f98c9503149fb606e8: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-1/wal/wals/f1310b3cca2048b3b02f4dc0c63adf1f
I20260928 08:48:42.810005 18050 ts_tablet_manager.cc:1939] T f1310b3cca2048b3b02f4dc0c63adf1f P 158749d76d0d44f98c9503149fb606e8: Deleting consensus metadata
I20260928 08:48:42.811385 17534 catalog_manager.cc:4841] TS c2e2c865320a4c5e9236d8b1613cdebc (127.0.97.1:38257): tablet f1310b3cca2048b3b02f4dc0c63adf1f (table test-table [id=c5d2b2aca1d44fd6a0d50b400953df73]) successfully deleted
I20260928 08:48:42.811792 17535 catalog_manager.cc:4841] TS 158749d76d0d44f98c9503149fb606e8 (127.0.97.2:44381): tablet f1310b3cca2048b3b02f4dc0c63adf1f (table test-table [id=c5d2b2aca1d44fd6a0d50b400953df73]) successfully deleted
I20260928 08:48:42.812870 18048 ts_tablet_manager.cc:1918] T f1310b3cca2048b3b02f4dc0c63adf1f P 8c27228bc47d4f4e80efdea7487a96d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:48:42.813155 18048 log.cc:1196] T f1310b3cca2048b3b02f4dc0c63adf1f P 8c27228bc47d4f4e80efdea7487a96d0: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790585022442112-388-0/minicluster-data/ts-2/wal/wals/f1310b3cca2048b3b02f4dc0c63adf1f
I20260928 08:48:42.813755 18048 ts_tablet_manager.cc:1939] T f1310b3cca2048b3b02f4dc0c63adf1f P 8c27228bc47d4f4e80efdea7487a96d0: Deleting consensus metadata
I20260928 08:48:42.815635 17533 catalog_manager.cc:4841] TS 8c27228bc47d4f4e80efdea7487a96d0 (127.0.97.3:39315): tablet f1310b3cca2048b3b02f4dc0c63adf1f (table test-table [id=c5d2b2aca1d44fd6a0d50b400953df73]) successfully deleted
Sep 28 08:48:42 dist-test-slave-n6xs krb5kdc[17504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:48:42.825692 18022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:42.791869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48826 (local address 127.0.97.1:38257)
0928 08:48:42.792127 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:42.792137 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:42.808424 (+ 16287us) server_negotiation.cc:407] Connection header received
0928 08:48:42.808946 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:42.808954 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:42.809064 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:42.809151 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:42.811366 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:42.812366 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:42.813943 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:42.814423 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:42.814575 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:42.819419 (+ 4844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:42.819433 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:42.819448 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:48:42.819492 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:42.822577 (+ 3085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:42.823363 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:42.823371 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:42.823374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:42.823521 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:42.824022 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:42.824030 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:42.824033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:42.824413 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:42.824764 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:42.825078 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 08:48:42.825316 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 08:48:42.826584 17733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:48:42.919502 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 17610
I20260928 08:48:42.946180 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 17744
Sep 28 08:48:42 dist-test-slave-n6xs krb5kdc[17504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 08:48:42.965214 18020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:42.947171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51180 (local address 127.0.97.2:44381)
0928 08:48:42.947362 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:42.947366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:42.947385 (+ 19us) server_negotiation.cc:407] Connection header received
0928 08:48:42.947697 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:42.947702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:42.947781 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:42.947941 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:42.949666 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:42.950439 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:42.952488 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:42.952884 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:42.952990 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:42.958404 (+ 5414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:42.958422 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:42.958438 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:48:42.958481 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:42.961643 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:42.962553 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:42.962561 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:42.962564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:42.962704 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:42.963300 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:42.963303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:42.963306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:42.963736 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:42.964104 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:42.964546 (+ 442us) server_negotiation.cc:299] Negotiation successful
0928 08:48:42.964810 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 08:48:42.965941 17867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:48:43.084117 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 17744
I20260928 08:48:43.105955 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 17878
Sep 28 08:48:43 dist-test-slave-n6xs krb5kdc[17504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 08:48:43.125008 18024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:43.107013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57136 (local address 127.0.97.3:39315)
0928 08:48:43.107237 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:43.107243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:43.107265 (+ 22us) server_negotiation.cc:407] Connection header received
0928 08:48:43.107334 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:43.107340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:43.107426 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:43.107546 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:43.108985 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:43.110217 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:43.111706 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:43.112460 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:43.112612 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:43.117354 (+ 4742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:43.117373 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:43.117393 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:48:43.117462 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:43.121719 (+ 4257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:43.122540 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:43.122551 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:43.122555 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:43.122762 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:43.123156 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:43.123161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:43.123164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:43.123687 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:43.124071 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:43.124349 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 08:48:43.124599 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 08:48:43.125921 18001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:48:43.249665 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 17878
I20260928 08:48:43.269335 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 17517
I20260928 08:48:43.270500 17579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:48:43.385661 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 17517
2026-09-28T08:48:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4675 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T08:48:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:48:43Z Disabled control of system clock
I20260928 08:48:43.467307 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33415
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:34803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 08:48:43.826925 18059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:48:43.827270 18059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:48:43.827354 18059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:48:43.835152 18059 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:48:43.835294 18059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:48:43.835333 18059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:48:43.835384 18059 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:48:43.835422 18059 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:48:43.845418 18059 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33415
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:48:43.846864 18059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:48:43.849010 18059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:48:43.862064 18067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:43.862046 18064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:43.862198 18065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:48:43.862875 18059 server_base.cc:1034] running on GCE node
I20260928 08:48:43.863716 18059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:48:43.865406 18059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:48:43.866650 18059 hybrid_clock.cc:648] HybridClock initialized: now 1790585323866591 us; error 51 us; skew 500 ppm
I20260928 08:48:43.867105 18059 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:48:43.876039 18059 webserver.cc:460] Webserver started at http://127.0.97.62:35295/ using document root <none> and password file <none>
I20260928 08:48:43.876976 18059 fs_manager.cc:359] Metadata directory not provided
I20260928 08:48:43.877110 18059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:48:43.877517 18059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:48:43.880991 18059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "8a178d190afa4f37abd669ea0b1151f3"
format_stamp: "Formatted at 2026-09-28 08:48:43 on dist-test-slave-n6xs"
server_key: "a1d061e2970f0900493aaa05b78f920c"
server_key_iv: "219f3c0f2fbc26375be55d0c6e6da32b"
server_key_version: "encryptionkey@0"
I20260928 08:48:43.881960 18059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "8a178d190afa4f37abd669ea0b1151f3"
format_stamp: "Formatted at 2026-09-28 08:48:43 on dist-test-slave-n6xs"
server_key: "a1d061e2970f0900493aaa05b78f920c"
server_key_iv: "219f3c0f2fbc26375be55d0c6e6da32b"
server_key_version: "encryptionkey@0"
I20260928 08:48:43.888635 18059 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260928 08:48:43.892867 18073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:43.894584 18059 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 08:48:43.894822 18059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "8a178d190afa4f37abd669ea0b1151f3"
format_stamp: "Formatted at 2026-09-28 08:48:43 on dist-test-slave-n6xs"
server_key: "a1d061e2970f0900493aaa05b78f920c"
server_key_iv: "219f3c0f2fbc26375be55d0c6e6da32b"
server_key_version: "encryptionkey@0"
I20260928 08:48:43.895073 18059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:48:43.917747 18059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:48:43.918763 18059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:48:43.919101 18059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:48:43.940248 18059 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33415
I20260928 08:48:43.940251 18124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33415 every 8 connection(s)
I20260928 08:48:43.941987 18059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:48:43.944576 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 18059
I20260928 08:48:43.944777 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:48:43.945276 388 external_mini_cluster.cc:1428] Setting key 8bfa4bc8bd25232a6310802f9da5b826
I20260928 08:48:43.947088 18125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:48:43.961447 18128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:43.947373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41178 (local address 127.0.97.62:33415)
0928 08:48:43.948536 (+ 1163us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:43.948548 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:43.948593 (+ 45us) server_negotiation.cc:407] Connection header received
0928 08:48:43.950262 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:43.950288 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:43.950897 (+ 609us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:43.951313 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:43.952109 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:43.953634 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:43.955929 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:43.956764 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:43.956938 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:43.958424 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:43.958433 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:43.958457 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:48:43.959221 (+ 764us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:43.959325 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:48:43.959355 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:43.959470 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:43.959770 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 08:48:43.960270 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":437,"threads_started":1}
I20260928 08:48:43.961869 18125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a178d190afa4f37abd669ea0b1151f3: Bootstrap starting.
I20260928 08:48:43.965838 18125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a178d190afa4f37abd669ea0b1151f3: Neither blocks nor log segments found. Creating new log.
I20260928 08:48:43.966964 18125 log.cc:824] T 00000000000000000000000000000000 P 8a178d190afa4f37abd669ea0b1151f3: Log is configured to *not* fsync() on all Append() calls
I20260928 08:48:43.971057 18125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a178d190afa4f37abd669ea0b1151f3: No bootstrap required, opened a new log
I20260928 08:48:43.979542 18125 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a178d190afa4f37abd669ea0b1151f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a178d190afa4f37abd669ea0b1151f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33415 } }
I20260928 08:48:43.980108 18125 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a178d190afa4f37abd669ea0b1151f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:48:43.980209 18125 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a178d190afa4f37abd669ea0b1151f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a178d190afa4f37abd669ea0b1151f3, State: Initialized, Role: FOLLOWER
I20260928 08:48:43.980873 18125 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a178d190afa4f37abd669ea0b1151f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a178d190afa4f37abd669ea0b1151f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33415 } }
I20260928 08:48:43.981113 18125 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a178d190afa4f37abd669ea0b1151f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:48:43.981288 18125 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a178d190afa4f37abd669ea0b1151f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:48:43.981472 18125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a178d190afa4f37abd669ea0b1151f3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:48:43.984143 18125 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a178d190afa4f37abd669ea0b1151f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a178d190afa4f37abd669ea0b1151f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33415 } }
I20260928 08:48:43.984719 18125 leader_election.cc:302] T 00000000000000000000000000000000 P 8a178d190afa4f37abd669ea0b1151f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a178d190afa4f37abd669ea0b1151f3; no voters:
I20260928 08:48:43.985337 18125 leader_election.cc:288] T 00000000000000000000000000000000 P 8a178d190afa4f37abd669ea0b1151f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:48:43.985606 18130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a178d190afa4f37abd669ea0b1151f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:48:43.986385 18130 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a178d190afa4f37abd669ea0b1151f3 [term 1 LEADER]: Becoming Leader. State: Replica: 8a178d190afa4f37abd669ea0b1151f3, State: Running, Role: LEADER
I20260928 08:48:43.986956 18130 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a178d190afa4f37abd669ea0b1151f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a178d190afa4f37abd669ea0b1151f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33415 } }
I20260928 08:48:43.987872 18125 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a178d190afa4f37abd669ea0b1151f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:48:43.989959 18131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a178d190afa4f37abd669ea0b1151f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a178d190afa4f37abd669ea0b1151f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a178d190afa4f37abd669ea0b1151f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33415 } } }
I20260928 08:48:43.990445 18131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a178d190afa4f37abd669ea0b1151f3 [sys.catalog]: This master's current role is: LEADER
I20260928 08:48:43.991019 18132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a178d190afa4f37abd669ea0b1151f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a178d190afa4f37abd669ea0b1151f3. Latest consensus state: current_term: 1 leader_uuid: "8a178d190afa4f37abd669ea0b1151f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a178d190afa4f37abd669ea0b1151f3" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33415 } } }
I20260928 08:48:43.991344 18132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a178d190afa4f37abd669ea0b1151f3 [sys.catalog]: This master's current role is: LEADER
I20260928 08:48:43.992332 18136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:48:43.997855 18136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:48:44.009100 18136 catalog_manager.cc:1347] Generated new cluster ID: 243c3ba8308b456588aa298a6558f030
I20260928 08:48:44.009276 18136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:48:44.024333 18136 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:48:44.026423 18136 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:48:44.036723 18136 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a178d190afa4f37abd669ea0b1151f3: Generated new TSK 0
I20260928 08:48:44.037729 18136 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:48:44.049793 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33415
--builtin_ntp_servers=127.0.97.20:34803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 08:48:44.444183 18149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:48:44.444499 18149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:48:44.444643 18149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:48:44.452455 18149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:48:44.452579 18149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:48:44.452744 18149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:48:44.465288 18149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33415
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:48:44.466779 18149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:48:44.468858 18149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:48:44.483144 18154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:44.483032 18157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:44.483057 18155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:48:44.483898 18149 server_base.cc:1034] running on GCE node
I20260928 08:48:44.484597 18149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:48:44.485618 18149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:48:44.486871 18149 hybrid_clock.cc:648] HybridClock initialized: now 1790585324486810 us; error 55 us; skew 500 ppm
I20260928 08:48:44.487306 18149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:48:44.490355 18149 webserver.cc:460] Webserver started at http://127.0.97.1:42243/ using document root <none> and password file <none>
I20260928 08:48:44.491132 18149 fs_manager.cc:359] Metadata directory not provided
I20260928 08:48:44.491252 18149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:48:44.491652 18149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:48:44.494773 18149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-0/data/instance:
uuid: "37f212074e104290951fd91fa590aa75"
format_stamp: "Formatted at 2026-09-28 08:48:44 on dist-test-slave-n6xs"
server_key: "cd8358cf5ee82e556b3f287020d05bb4"
server_key_iv: "f6da10291410678e47f9095231f47207"
server_key_version: "encryptionkey@0"
I20260928 08:48:44.495638 18149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-0/wal/instance:
uuid: "37f212074e104290951fd91fa590aa75"
format_stamp: "Formatted at 2026-09-28 08:48:44 on dist-test-slave-n6xs"
server_key: "cd8358cf5ee82e556b3f287020d05bb4"
server_key_iv: "f6da10291410678e47f9095231f47207"
server_key_version: "encryptionkey@0"
I20260928 08:48:44.501610 18149 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260928 08:48:44.505652 18163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:44.507220 18149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:48:44.507424 18149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-0/wal
uuid: "37f212074e104290951fd91fa590aa75"
format_stamp: "Formatted at 2026-09-28 08:48:44 on dist-test-slave-n6xs"
server_key: "cd8358cf5ee82e556b3f287020d05bb4"
server_key_iv: "f6da10291410678e47f9095231f47207"
server_key_version: "encryptionkey@0"
I20260928 08:48:44.507673 18149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:48:44.535771 18149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:48:44.536763 18149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:48:44.537117 18149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:48:44.538165 18149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:48:44.539930 18149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:48:44.540055 18149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:44.540197 18149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:48:44.540284 18149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:44.575030 18149 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:45617
I20260928 08:48:44.575129 18275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:45617 every 8 connection(s)
I20260928 08:48:44.576460 18149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:48:44.582206 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 18149
I20260928 08:48:44.582346 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:48:44.582792 388 external_mini_cluster.cc:1428] Setting key e7a972e574c2047f4115025a0afa719e
I20260928 08:48:44.586072 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33415
--builtin_ntp_servers=127.0.97.20:34803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 08:48:44.592084 18278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:44.579371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:60335 (local address 127.0.97.62:33415)
0928 08:48:44.579928 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:44.579934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:44.581317 (+ 1383us) server_negotiation.cc:407] Connection header received
0928 08:48:44.583506 (+ 2189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:44.583512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:44.583626 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:44.583773 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:44.585412 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:44.586541 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:44.588466 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:44.589274 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:44.589421 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:44.590980 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:44.590990 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:44.590998 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:48:44.591030 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:44.591089 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:48:44.591108 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:44.591201 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:44.591446 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 08:48:44.591573 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":236,"threads_started":1}
I20260928 08:48:44.593084 18279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:44.580067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33415 (local address 127.0.97.1:60335)
0928 08:48:44.581103 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:44.581165 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:44.583090 (+ 1925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:44.584163 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:44.584176 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:44.584313 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:48:44.585218 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:44.585239 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:44.586724 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:44.586729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:44.588306 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:44.588320 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:44.589540 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:44.589545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:44.589769 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:44.590572 (+ 803us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:48:44.590575 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:48:44.590626 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:48:44.590717 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:48:44.591208 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:48:44.591216 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:48:44.591222 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 08:48:44.591381 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 08:48:44.591778 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":339,"threads_started":1}
I20260928 08:48:44.595021 18276 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33415
I20260928 08:48:44.595403 18276 heartbeater.cc:464] Registering TS with master...
I20260928 08:48:44.596436 18276 heartbeater.cc:511] Master 127.0.97.62:33415 requested a full tablet report, sending...
I20260928 08:48:44.599174 18090 ts_manager.cc:194] Registered new tserver with Master: 37f212074e104290951fd91fa590aa75 (127.0.97.1:45617)
I20260928 08:48:44.600940 18090 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:60335
W20260928 08:48:44.941975 18280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:48:44.942361 18280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:48:44.942543 18280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:48:44.951061 18280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:48:44.951268 18280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:48:44.951498 18280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:48:44.962232 18280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33415
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:48:44.963696 18280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:48:44.965781 18280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:48:44.981521 18285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:44.981488 18286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:44.981452 18288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:48:44.982491 18280 server_base.cc:1034] running on GCE node
I20260928 08:48:44.983286 18280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:48:44.984545 18280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:48:44.985854 18280 hybrid_clock.cc:648] HybridClock initialized: now 1790585324985793 us; error 65 us; skew 500 ppm
I20260928 08:48:44.986285 18280 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:48:44.989370 18280 webserver.cc:460] Webserver started at http://127.0.97.2:39693/ using document root <none> and password file <none>
I20260928 08:48:44.990302 18280 fs_manager.cc:359] Metadata directory not provided
I20260928 08:48:44.990432 18280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:48:44.990742 18280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:48:44.994187 18280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-1/data/instance:
uuid: "5746d762f1ba492490b915617bb8b5c7"
format_stamp: "Formatted at 2026-09-28 08:48:44 on dist-test-slave-n6xs"
server_key: "0d2e66fdd17e2e046a7accd1d6610913"
server_key_iv: "ed6d067210180b6b50d1e91d4ab2459f"
server_key_version: "encryptionkey@0"
I20260928 08:48:44.995216 18280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-1/wal/instance:
uuid: "5746d762f1ba492490b915617bb8b5c7"
format_stamp: "Formatted at 2026-09-28 08:48:44 on dist-test-slave-n6xs"
server_key: "0d2e66fdd17e2e046a7accd1d6610913"
server_key_iv: "ed6d067210180b6b50d1e91d4ab2459f"
server_key_version: "encryptionkey@0"
I20260928 08:48:45.002170 18280 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260928 08:48:45.006837 18294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:45.008688 18280 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 08:48:45.008910 18280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-1/wal
uuid: "5746d762f1ba492490b915617bb8b5c7"
format_stamp: "Formatted at 2026-09-28 08:48:44 on dist-test-slave-n6xs"
server_key: "0d2e66fdd17e2e046a7accd1d6610913"
server_key_iv: "ed6d067210180b6b50d1e91d4ab2459f"
server_key_version: "encryptionkey@0"
I20260928 08:48:45.009189 18280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:48:45.028626 18280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:48:45.029593 18280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:48:45.029917 18280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:48:45.030987 18280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:48:45.032661 18280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:48:45.032752 18280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:45.032925 18280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:48:45.033022 18280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:45.069271 18280 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:40241
I20260928 08:48:45.069321 18406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:40241 every 8 connection(s)
I20260928 08:48:45.070814 18280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:48:45.077272 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 18280
I20260928 08:48:45.077476 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:48:45.078068 388 external_mini_cluster.cc:1428] Setting key 27044cd7fb54042e4050e6fbfc4b2339
I20260928 08:48:45.081204 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33415
--builtin_ntp_servers=127.0.97.20:34803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 08:48:45.085685 18278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:45.073746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:42353 (local address 127.0.97.62:33415)
0928 08:48:45.073932 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:45.073936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:45.075446 (+ 1510us) server_negotiation.cc:407] Connection header received
0928 08:48:45.077598 (+ 2152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:45.077607 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:45.077713 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:45.077838 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:45.079278 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:45.080223 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:45.082119 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:45.082912 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:45.083069 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:45.084673 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:45.084683 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:45.084691 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:48:45.084720 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:45.084770 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:48:45.084789 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:45.084889 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:45.085130 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 08:48:45.085263 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 08:48:45.086714 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:45.074224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33415 (local address 127.0.97.2:42353)
0928 08:48:45.075258 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:45.075297 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:45.077167 (+ 1870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:45.078223 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:45.078238 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:45.078370 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:48:45.079072 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:45.079092 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:45.080445 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:45.080450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:45.081927 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:45.081943 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:45.083168 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:45.083173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:45.083428 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:45.084263 (+ 835us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:48:45.084266 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:48:45.084313 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:48:45.084413 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:48:45.084893 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:48:45.084901 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:48:45.084907 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 08:48:45.085075 (+ 168us) client_negotiation.cc:240] Negotiation successful
0928 08:48:45.085432 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":328,"threads_started":1}
I20260928 08:48:45.088457 18407 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33415
I20260928 08:48:45.088801 18407 heartbeater.cc:464] Registering TS with master...
I20260928 08:48:45.089741 18407 heartbeater.cc:511] Master 127.0.97.62:33415 requested a full tablet report, sending...
I20260928 08:48:45.091427 18090 ts_manager.cc:194] Registered new tserver with Master: 5746d762f1ba492490b915617bb8b5c7 (127.0.97.2:40241)
I20260928 08:48:45.092363 18090 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:42353
W20260928 08:48:45.440310 18410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:48:45.440634 18410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:48:45.440783 18410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:48:45.448648 18410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:48:45.448767 18410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:48:45.448941 18410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:48:45.458570 18410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:34803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:33415
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:48:45.460047 18410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:48:45.461884 18410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:48:45.476172 18415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:45.476066 18418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:45.476107 18416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:48:45.477187 18410 server_base.cc:1034] running on GCE node
I20260928 08:48:45.477945 18410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:48:45.479234 18410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:48:45.480557 18410 hybrid_clock.cc:648] HybridClock initialized: now 1790585325480497 us; error 57 us; skew 500 ppm
I20260928 08:48:45.480887 18410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:48:45.483469 18410 webserver.cc:460] Webserver started at http://127.0.97.3:40865/ using document root <none> and password file <none>
I20260928 08:48:45.484494 18410 fs_manager.cc:359] Metadata directory not provided
I20260928 08:48:45.484632 18410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:48:45.485024 18410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:48:45.489952 18410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-2/data/instance:
uuid: "3d1afab0c50946919bdaaef6e2c99df5"
format_stamp: "Formatted at 2026-09-28 08:48:45 on dist-test-slave-n6xs"
server_key: "6b62fc58567f23d2b272a921d0974dd0"
server_key_iv: "eee81bce0da4cc105200b47f2d87efa9"
server_key_version: "encryptionkey@0"
I20260928 08:48:45.491405 18410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-2/wal/instance:
uuid: "3d1afab0c50946919bdaaef6e2c99df5"
format_stamp: "Formatted at 2026-09-28 08:48:45 on dist-test-slave-n6xs"
server_key: "6b62fc58567f23d2b272a921d0974dd0"
server_key_iv: "eee81bce0da4cc105200b47f2d87efa9"
server_key_version: "encryptionkey@0"
I20260928 08:48:45.497363 18410 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 08:48:45.501739 18424 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:45.503334 18410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:48:45.503645 18410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-2/wal
uuid: "3d1afab0c50946919bdaaef6e2c99df5"
format_stamp: "Formatted at 2026-09-28 08:48:45 on dist-test-slave-n6xs"
server_key: "6b62fc58567f23d2b272a921d0974dd0"
server_key_iv: "eee81bce0da4cc105200b47f2d87efa9"
server_key_version: "encryptionkey@0"
I20260928 08:48:45.503979 18410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:48:45.522003 18410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:48:45.522966 18410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:48:45.523276 18410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:48:45.524412 18410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:48:45.526046 18410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:48:45.526154 18410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:45.526324 18410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:48:45.526407 18410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:45.561621 18410 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:34491
I20260928 08:48:45.561679 18536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:34491 every 8 connection(s)
I20260928 08:48:45.563064 18410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:48:45.570523 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 18410
I20260928 08:48:45.570739 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:48:45.571198 388 external_mini_cluster.cc:1428] Setting key 4148d6727c5509f89858830bfabd67fa
I20260928 08:48:45.576821 18278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:45.565833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47171 (local address 127.0.97.62:33415)
0928 08:48:45.566041 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:45.566046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:45.567478 (+ 1432us) server_negotiation.cc:407] Connection header received
0928 08:48:45.569435 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:45.569442 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:45.569551 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:45.569673 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:45.570898 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:45.571724 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:45.573557 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:45.574154 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:45.574292 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:45.575910 (+ 1618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:45.575918 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:45.575923 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:48:45.575956 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:45.576000 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:48:45.576017 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:45.576111 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:45.576356 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 08:48:45.576483 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 08:48:45.577818 18539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:45.566342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33415 (local address 127.0.97.3:47171)
0928 08:48:45.567282 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:45.567324 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:45.569112 (+ 1788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:45.569922 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:45.569931 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:45.570034 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:48:45.570667 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:45.570712 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:45.571917 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:45.571923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:45.573377 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:45.573391 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:45.574367 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:45.574373 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:45.574624 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:45.575472 (+ 848us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:48:45.575475 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:48:45.575519 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:48:45.575636 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:48:45.576123 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:48:45.576131 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:48:45.576138 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 08:48:45.576312 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 08:48:45.576673 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":297,"threads_started":1}
I20260928 08:48:45.579468 18537 heartbeater.cc:347] Connected to a master server at 127.0.97.62:33415
I20260928 08:48:45.579880 18537 heartbeater.cc:464] Registering TS with master...
I20260928 08:48:45.580665 18537 heartbeater.cc:511] Master 127.0.97.62:33415 requested a full tablet report, sending...
I20260928 08:48:45.582260 18090 ts_manager.cc:194] Registered new tserver with Master: 3d1afab0c50946919bdaaef6e2c99df5 (127.0.97.3:34491)
I20260928 08:48:45.583091 18090 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:47171
I20260928 08:48:45.586643 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:48:45.597957 18278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:45.591513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41186 (local address 127.0.97.62:33415)
0928 08:48:45.591798 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:45.591825 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:45.592109 (+ 284us) server_negotiation.cc:407] Connection header received
0928 08:48:45.592279 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:45.592284 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:45.592352 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:45.592445 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:45.593049 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:45.593804 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:45.595369 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:45.595905 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:45.596025 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:45.597134 (+ 1109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:45.597138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:45.597141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:48:45.597162 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:45.597209 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:48:45.597220 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:45.597294 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:45.597512 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 08:48:45.597598 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 08:48:45.605268 18276 heartbeater.cc:503] Master 127.0.97.62:33415 was elected leader, sending a full tablet report...
I20260928 08:48:45.602615 18090 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:48:45.610814 18090 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:48:45.647992 18549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:45.625147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34491 (local address 127.0.0.1:50394)
0928 08:48:45.632547 (+ 7400us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:45.632613 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:45.632841 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:45.639745 (+ 6904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:45.639751 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:45.639783 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:48:45.640275 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:45.640290 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:45.642366 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:45.642372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:45.644890 (+ 2518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:45.644907 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:45.647025 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:45.647034 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:45.647217 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:45.647248 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 08:48:45.647341 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 08:48:45.647500 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6945,"mutex_wait_us":81,"thread_start_us":468,"threads_started":1}
I20260928 08:48:45.648057 18550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:45.625415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50394 (local address 127.0.97.3:34491)
0928 08:48:45.639084 (+ 13669us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:45.639093 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:45.639129 (+ 36us) server_negotiation.cc:407] Connection header received
0928 08:48:45.639222 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:45.639230 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:45.639496 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:45.640407 (+ 911us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:48:45.640682 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:45.642192 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:45.645118 (+ 2926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:45.646747 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:45.646895 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:45.647185 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:45.647428 (+ 243us) server_negotiation.cc:299] Negotiation successful
0928 08:48:45.647627 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13470,"thread_start_us":13302,"threads_started":1}
I20260928 08:48:45.648223 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:45.626035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54432 (local address 127.0.97.2:40241)
0928 08:48:45.637160 (+ 11125us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:45.637172 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:45.638156 (+ 984us) server_negotiation.cc:407] Connection header received
0928 08:48:45.638277 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:45.638287 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:45.638638 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:45.638871 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:48:45.640049 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:45.641559 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:45.644433 (+ 2874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:45.646122 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:45.646264 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:45.646554 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:45.647643 (+ 1089us) server_negotiation.cc:299] Negotiation successful
0928 08:48:45.647859 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10929,"thread_start_us":11122,"threads_started":1}
I20260928 08:48:45.648504 18554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:45.627530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58142 (local address 127.0.97.1:45617)
0928 08:48:45.639505 (+ 11975us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:45.639517 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:45.640282 (+ 765us) server_negotiation.cc:407] Connection header received
0928 08:48:45.640545 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:45.640554 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:45.640810 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:45.641024 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:48:45.641732 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:45.643379 (+ 1647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:45.645834 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:45.646874 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:45.647024 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:45.647309 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:45.647922 (+ 613us) server_negotiation.cc:299] Negotiation successful
0928 08:48:45.648145 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11761,"thread_start_us":11612,"threads_started":1}
I20260928 08:48:45.649041 18553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:45.626778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:45617 (local address 127.0.0.1:58142)
0928 08:48:45.640132 (+ 13354us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:45.640152 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:45.640337 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:45.641051 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:45.641056 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:45.641087 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:48:45.641517 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:45.641530 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:45.643564 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:45.643569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:45.645648 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:45.645659 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:45.647448 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:45.647454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:45.647680 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:45.647714 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 08:48:45.647800 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 08:48:45.648668 (+ 868us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13223,"mutex_wait_us":1,"thread_start_us":13020,"threads_started":1}
I20260928 08:48:45.649784 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:45.625778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40241 (local address 127.0.0.1:54432)
0928 08:48:45.637848 (+ 12070us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:45.637868 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:45.638004 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:45.639182 (+ 1178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:45.639191 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:45.639226 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:48:45.639720 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:45.639736 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:45.641708 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:45.641716 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:45.644128 (+ 2412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:45.644141 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:45.646690 (+ 2549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:45.646698 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:45.646943 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:45.647011 (+ 68us) client_negotiation.cc:769] Sending connection context
0928 08:48:45.647153 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 08:48:45.647398 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11944,"mutex_wait_us":48,"thread_start_us":12534,"threads_started":1}
I20260928 08:48:45.652930 18211 tablet_service.cc:1467] Processing CreateTablet for tablet 516b1021c8514f28b815929750f049da (DEFAULT_TABLE table=test-table [id=44d8ac381ba64ecb8a84dd202d2a3e9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:48:45.653100 18472 tablet_service.cc:1467] Processing CreateTablet for tablet 516b1021c8514f28b815929750f049da (DEFAULT_TABLE table=test-table [id=44d8ac381ba64ecb8a84dd202d2a3e9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:48:45.653710 18342 tablet_service.cc:1467] Processing CreateTablet for tablet 516b1021c8514f28b815929750f049da (DEFAULT_TABLE table=test-table [id=44d8ac381ba64ecb8a84dd202d2a3e9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:48:45.654444 18211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 516b1021c8514f28b815929750f049da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:48:45.654573 18472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 516b1021c8514f28b815929750f049da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:48:45.655011 18342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 516b1021c8514f28b815929750f049da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:48:45.666414 18555 tablet_bootstrap.cc:492] T 516b1021c8514f28b815929750f049da P 37f212074e104290951fd91fa590aa75: Bootstrap starting.
I20260928 08:48:45.670210 18555 tablet_bootstrap.cc:654] T 516b1021c8514f28b815929750f049da P 37f212074e104290951fd91fa590aa75: Neither blocks nor log segments found. Creating new log.
I20260928 08:48:45.671483 18556 tablet_bootstrap.cc:492] T 516b1021c8514f28b815929750f049da P 5746d762f1ba492490b915617bb8b5c7: Bootstrap starting.
I20260928 08:48:45.671646 18555 log.cc:824] T 516b1021c8514f28b815929750f049da P 37f212074e104290951fd91fa590aa75: Log is configured to *not* fsync() on all Append() calls
I20260928 08:48:45.672559 18557 tablet_bootstrap.cc:492] T 516b1021c8514f28b815929750f049da P 3d1afab0c50946919bdaaef6e2c99df5: Bootstrap starting.
I20260928 08:48:45.675503 18556 tablet_bootstrap.cc:654] T 516b1021c8514f28b815929750f049da P 5746d762f1ba492490b915617bb8b5c7: Neither blocks nor log segments found. Creating new log.
I20260928 08:48:45.676122 18557 tablet_bootstrap.cc:654] T 516b1021c8514f28b815929750f049da P 3d1afab0c50946919bdaaef6e2c99df5: Neither blocks nor log segments found. Creating new log.
I20260928 08:48:45.676648 18555 tablet_bootstrap.cc:492] T 516b1021c8514f28b815929750f049da P 37f212074e104290951fd91fa590aa75: No bootstrap required, opened a new log
I20260928 08:48:45.676862 18556 log.cc:824] T 516b1021c8514f28b815929750f049da P 5746d762f1ba492490b915617bb8b5c7: Log is configured to *not* fsync() on all Append() calls
I20260928 08:48:45.677013 18555 ts_tablet_manager.cc:1397] T 516b1021c8514f28b815929750f049da P 37f212074e104290951fd91fa590aa75: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20260928 08:48:45.677450 18557 log.cc:824] T 516b1021c8514f28b815929750f049da P 3d1afab0c50946919bdaaef6e2c99df5: Log is configured to *not* fsync() on all Append() calls
I20260928 08:48:45.680603 18556 tablet_bootstrap.cc:492] T 516b1021c8514f28b815929750f049da P 5746d762f1ba492490b915617bb8b5c7: No bootstrap required, opened a new log
I20260928 08:48:45.680923 18556 ts_tablet_manager.cc:1397] T 516b1021c8514f28b815929750f049da P 5746d762f1ba492490b915617bb8b5c7: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260928 08:48:45.681432 18557 tablet_bootstrap.cc:492] T 516b1021c8514f28b815929750f049da P 3d1afab0c50946919bdaaef6e2c99df5: No bootstrap required, opened a new log
I20260928 08:48:45.681715 18557 ts_tablet_manager.cc:1397] T 516b1021c8514f28b815929750f049da P 3d1afab0c50946919bdaaef6e2c99df5: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 08:48:45.688484 18555 raft_consensus.cc:348] T 516b1021c8514f28b815929750f049da P 37f212074e104290951fd91fa590aa75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1afab0c50946919bdaaef6e2c99df5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34491 } } peers { permanent_uuid: "37f212074e104290951fd91fa590aa75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45617 } } peers { permanent_uuid: "5746d762f1ba492490b915617bb8b5c7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40241 } }
I20260928 08:48:45.688484 18556 raft_consensus.cc:348] T 516b1021c8514f28b815929750f049da P 5746d762f1ba492490b915617bb8b5c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1afab0c50946919bdaaef6e2c99df5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34491 } } peers { permanent_uuid: "37f212074e104290951fd91fa590aa75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45617 } } peers { permanent_uuid: "5746d762f1ba492490b915617bb8b5c7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40241 } }
I20260928 08:48:45.688993 18555 raft_consensus.cc:374] T 516b1021c8514f28b815929750f049da P 37f212074e104290951fd91fa590aa75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:48:45.689007 18556 raft_consensus.cc:374] T 516b1021c8514f28b815929750f049da P 5746d762f1ba492490b915617bb8b5c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:48:45.689114 18556 raft_consensus.cc:729] T 516b1021c8514f28b815929750f049da P 5746d762f1ba492490b915617bb8b5c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5746d762f1ba492490b915617bb8b5c7, State: Initialized, Role: FOLLOWER
I20260928 08:48:45.689134 18555 raft_consensus.cc:729] T 516b1021c8514f28b815929750f049da P 37f212074e104290951fd91fa590aa75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37f212074e104290951fd91fa590aa75, State: Initialized, Role: FOLLOWER
I20260928 08:48:45.689836 18555 consensus_queue.cc:260] T 516b1021c8514f28b815929750f049da P 37f212074e104290951fd91fa590aa75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1afab0c50946919bdaaef6e2c99df5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34491 } } peers { permanent_uuid: "37f212074e104290951fd91fa590aa75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45617 } } peers { permanent_uuid: "5746d762f1ba492490b915617bb8b5c7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40241 } }
I20260928 08:48:45.689910 18556 consensus_queue.cc:260] T 516b1021c8514f28b815929750f049da P 5746d762f1ba492490b915617bb8b5c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1afab0c50946919bdaaef6e2c99df5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34491 } } peers { permanent_uuid: "37f212074e104290951fd91fa590aa75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45617 } } peers { permanent_uuid: "5746d762f1ba492490b915617bb8b5c7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40241 } }
I20260928 08:48:45.692646 18407 heartbeater.cc:503] Master 127.0.97.62:33415 was elected leader, sending a full tablet report...
I20260928 08:48:45.693745 18557 raft_consensus.cc:348] T 516b1021c8514f28b815929750f049da P 3d1afab0c50946919bdaaef6e2c99df5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1afab0c50946919bdaaef6e2c99df5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34491 } } peers { permanent_uuid: "37f212074e104290951fd91fa590aa75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45617 } } peers { permanent_uuid: "5746d762f1ba492490b915617bb8b5c7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40241 } }
I20260928 08:48:45.694376 18557 raft_consensus.cc:374] T 516b1021c8514f28b815929750f049da P 3d1afab0c50946919bdaaef6e2c99df5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:48:45.694500 18557 raft_consensus.cc:729] T 516b1021c8514f28b815929750f049da P 3d1afab0c50946919bdaaef6e2c99df5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d1afab0c50946919bdaaef6e2c99df5, State: Initialized, Role: FOLLOWER
I20260928 08:48:45.694880 18556 ts_tablet_manager.cc:1428] T 516b1021c8514f28b815929750f049da P 5746d762f1ba492490b915617bb8b5c7: Time spent starting tablet: real 0.014s user 0.007s sys 0.005s
I20260928 08:48:45.695472 18557 consensus_queue.cc:260] T 516b1021c8514f28b815929750f049da P 3d1afab0c50946919bdaaef6e2c99df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1afab0c50946919bdaaef6e2c99df5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34491 } } peers { permanent_uuid: "37f212074e104290951fd91fa590aa75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45617 } } peers { permanent_uuid: "5746d762f1ba492490b915617bb8b5c7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40241 } }
I20260928 08:48:45.696883 18537 heartbeater.cc:503] Master 127.0.97.62:33415 was elected leader, sending a full tablet report...
I20260928 08:48:45.697649 18557 ts_tablet_manager.cc:1428] T 516b1021c8514f28b815929750f049da P 3d1afab0c50946919bdaaef6e2c99df5: Time spent starting tablet: real 0.016s user 0.008s sys 0.008s
I20260928 08:48:45.698217 18555 ts_tablet_manager.cc:1428] T 516b1021c8514f28b815929750f049da P 37f212074e104290951fd91fa590aa75: Time spent starting tablet: real 0.021s user 0.014s sys 0.006s
W20260928 08:48:45.814966 18538 tablet.cc:2369] T 516b1021c8514f28b815929750f049da P 3d1afab0c50946919bdaaef6e2c99df5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:48:45.823163 18408 tablet.cc:2369] T 516b1021c8514f28b815929750f049da P 5746d762f1ba492490b915617bb8b5c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:48:45.829201 18277 tablet.cc:2369] T 516b1021c8514f28b815929750f049da P 37f212074e104290951fd91fa590aa75: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:48:46.062175 18562 raft_consensus.cc:482] T 516b1021c8514f28b815929750f049da P 37f212074e104290951fd91fa590aa75 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:48:46.062528 18562 raft_consensus.cc:504] T 516b1021c8514f28b815929750f049da P 37f212074e104290951fd91fa590aa75 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1afab0c50946919bdaaef6e2c99df5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34491 } } peers { permanent_uuid: "37f212074e104290951fd91fa590aa75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45617 } } peers { permanent_uuid: "5746d762f1ba492490b915617bb8b5c7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40241 } }
I20260928 08:48:46.064393 18562 leader_election.cc:288] T 516b1021c8514f28b815929750f049da P 37f212074e104290951fd91fa590aa75 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d1afab0c50946919bdaaef6e2c99df5 (127.0.97.3:34491), 5746d762f1ba492490b915617bb8b5c7 (127.0.97.2:40241)
I20260928 08:48:46.073345 18550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:46.064995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:40983 (local address 127.0.97.3:34491)
0928 08:48:46.065226 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:46.065234 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:46.065885 (+ 651us) server_negotiation.cc:407] Connection header received
0928 08:48:46.065969 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:46.065974 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:46.066084 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:46.066189 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:48:46.067001 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:46.068151 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:46.071533 (+ 3382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:46.072380 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:46.072533 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:46.072641 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:46.072954 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 08:48:46.073082 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 08:48:46.073732 18565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:46.064881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:34491 (local address 127.0.97.1:40983)
0928 08:48:46.065741 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:46.065757 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:46.065862 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:46.066248 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:46.066253 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:46.066353 (+ 100us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:48:46.066808 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:46.066816 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:46.068311 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:46.068317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:46.071242 (+ 2925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:46.071260 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:46.072585 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:46.072592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:46.072765 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:46.072792 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 08:48:46.072926 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 08:48:46.073078 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":738,"mutex_wait_us":256,"spinlock_wait_cycles":25856,"thread_start_us":518,"threads_started":1}
I20260928 08:48:46.073987 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:46.064970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:41233 (local address 127.0.97.2:40241)
0928 08:48:46.065218 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:46.065228 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:46.065736 (+ 508us) server_negotiation.cc:407] Connection header received
0928 08:48:46.065971 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:46.065976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:46.066086 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:46.066179 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:48:46.067000 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:46.068151 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:46.071500 (+ 3349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:46.072784 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:46.072917 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:46.073043 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:46.073617 (+ 574us) server_negotiation.cc:299] Negotiation successful
0928 08:48:46.073751 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 08:48:46.074689 18564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:46.064764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:40241 (local address 127.0.97.1:41233)
0928 08:48:46.065597 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:46.065618 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:46.065804 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:46.066248 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:46.066253 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:46.066291 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:48:46.066800 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:46.066814 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:46.068313 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:46.068317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:46.071239 (+ 2922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:46.071260 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:46.073219 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:46.073228 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:46.073426 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:46.073460 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 08:48:46.074190 (+ 730us) client_negotiation.cc:240] Negotiation successful
0928 08:48:46.074346 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":228,"thread_start_us":464,"threads_started":1}
I20260928 08:48:46.074848 18492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516b1021c8514f28b815929750f049da" candidate_uuid: "37f212074e104290951fd91fa590aa75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1afab0c50946919bdaaef6e2c99df5" is_pre_election: true
I20260928 08:48:46.075446 18492 raft_consensus.cc:2454] T 516b1021c8514f28b815929750f049da P 3d1afab0c50946919bdaaef6e2c99df5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37f212074e104290951fd91fa590aa75 in term 0.
I20260928 08:48:46.075758 18362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516b1021c8514f28b815929750f049da" candidate_uuid: "37f212074e104290951fd91fa590aa75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5746d762f1ba492490b915617bb8b5c7" is_pre_election: true
I20260928 08:48:46.076177 18164 leader_election.cc:302] T 516b1021c8514f28b815929750f049da P 37f212074e104290951fd91fa590aa75 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37f212074e104290951fd91fa590aa75, 3d1afab0c50946919bdaaef6e2c99df5; no voters:
I20260928 08:48:46.076318 18362 raft_consensus.cc:2454] T 516b1021c8514f28b815929750f049da P 5746d762f1ba492490b915617bb8b5c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37f212074e104290951fd91fa590aa75 in term 0.
I20260928 08:48:46.076690 18562 raft_consensus.cc:2781] T 516b1021c8514f28b815929750f049da P 37f212074e104290951fd91fa590aa75 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:48:46.076869 18562 raft_consensus.cc:482] T 516b1021c8514f28b815929750f049da P 37f212074e104290951fd91fa590aa75 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:48:46.077003 18562 raft_consensus.cc:3037] T 516b1021c8514f28b815929750f049da P 37f212074e104290951fd91fa590aa75 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:48:46.079279 18562 raft_consensus.cc:504] T 516b1021c8514f28b815929750f049da P 37f212074e104290951fd91fa590aa75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1afab0c50946919bdaaef6e2c99df5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34491 } } peers { permanent_uuid: "37f212074e104290951fd91fa590aa75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45617 } } peers { permanent_uuid: "5746d762f1ba492490b915617bb8b5c7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40241 } }
I20260928 08:48:46.080088 18562 leader_election.cc:288] T 516b1021c8514f28b815929750f049da P 37f212074e104290951fd91fa590aa75 [CANDIDATE]: Term 1 election: Requested vote from peers 3d1afab0c50946919bdaaef6e2c99df5 (127.0.97.3:34491), 5746d762f1ba492490b915617bb8b5c7 (127.0.97.2:40241)
I20260928 08:48:46.080818 18492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516b1021c8514f28b815929750f049da" candidate_uuid: "37f212074e104290951fd91fa590aa75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d1afab0c50946919bdaaef6e2c99df5"
I20260928 08:48:46.080916 18362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "516b1021c8514f28b815929750f049da" candidate_uuid: "37f212074e104290951fd91fa590aa75" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5746d762f1ba492490b915617bb8b5c7"
I20260928 08:48:46.081060 18492 raft_consensus.cc:3037] T 516b1021c8514f28b815929750f049da P 3d1afab0c50946919bdaaef6e2c99df5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:48:46.081177 18362 raft_consensus.cc:3037] T 516b1021c8514f28b815929750f049da P 5746d762f1ba492490b915617bb8b5c7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:48:46.084316 18492 raft_consensus.cc:2454] T 516b1021c8514f28b815929750f049da P 3d1afab0c50946919bdaaef6e2c99df5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37f212074e104290951fd91fa590aa75 in term 1.
I20260928 08:48:46.084597 18362 raft_consensus.cc:2454] T 516b1021c8514f28b815929750f049da P 5746d762f1ba492490b915617bb8b5c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37f212074e104290951fd91fa590aa75 in term 1.
I20260928 08:48:46.084824 18164 leader_election.cc:302] T 516b1021c8514f28b815929750f049da P 37f212074e104290951fd91fa590aa75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37f212074e104290951fd91fa590aa75, 3d1afab0c50946919bdaaef6e2c99df5; no voters:
I20260928 08:48:46.085234 18562 raft_consensus.cc:2781] T 516b1021c8514f28b815929750f049da P 37f212074e104290951fd91fa590aa75 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:48:46.085642 18562 raft_consensus.cc:686] T 516b1021c8514f28b815929750f049da P 37f212074e104290951fd91fa590aa75 [term 1 LEADER]: Becoming Leader. State: Replica: 37f212074e104290951fd91fa590aa75, State: Running, Role: LEADER
I20260928 08:48:46.086208 18562 consensus_queue.cc:237] T 516b1021c8514f28b815929750f049da P 37f212074e104290951fd91fa590aa75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1afab0c50946919bdaaef6e2c99df5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34491 } } peers { permanent_uuid: "37f212074e104290951fd91fa590aa75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45617 } } peers { permanent_uuid: "5746d762f1ba492490b915617bb8b5c7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40241 } }
I20260928 08:48:46.091341 18089 catalog_manager.cc:5495] T 516b1021c8514f28b815929750f049da P 37f212074e104290951fd91fa590aa75 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37f212074e104290951fd91fa590aa75 (127.0.97.1). New cstate: current_term: 1 leader_uuid: "37f212074e104290951fd91fa590aa75" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d1afab0c50946919bdaaef6e2c99df5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37f212074e104290951fd91fa590aa75" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 45617 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5746d762f1ba492490b915617bb8b5c7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40241 } health_report { overall_health: UNKNOWN } } }
I20260928 08:48:46.207630 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:46.201122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58156 (local address 127.0.97.1:45617)
0928 08:48:46.201623 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:46.201629 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:46.201752 (+ 123us) server_negotiation.cc:407] Connection header received
0928 08:48:46.202017 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:46.202021 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:46.202107 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:46.202265 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:48:46.202764 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:46.203522 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:46.205231 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:46.205672 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:46.205777 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:46.206344 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:48:46.206913 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:48:46.207034 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:46.207179 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 08:48:46.207284 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":305,"threads_started":1}
I20260928 08:48:46.216540 18362 raft_consensus.cc:1261] T 516b1021c8514f28b815929750f049da P 5746d762f1ba492490b915617bb8b5c7 [term 1 FOLLOWER]: Refusing update from remote peer 37f212074e104290951fd91fa590aa75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:48:46.216542 18492 raft_consensus.cc:1261] T 516b1021c8514f28b815929750f049da P 3d1afab0c50946919bdaaef6e2c99df5 [term 1 FOLLOWER]: Refusing update from remote peer 37f212074e104290951fd91fa590aa75: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:48:46.217733 18562 consensus_queue.cc:1035] T 516b1021c8514f28b815929750f049da P 37f212074e104290951fd91fa590aa75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5746d762f1ba492490b915617bb8b5c7" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 40241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:48:46.218093 18566 consensus_queue.cc:1035] T 516b1021c8514f28b815929750f049da P 37f212074e104290951fd91fa590aa75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d1afab0c50946919bdaaef6e2c99df5" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 34491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:48:46.244765 18089 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:48:46.245091 18089 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:48:46.249058 18089 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a178d190afa4f37abd669ea0b1151f3: Sending DeleteTablet for 3 replicas of tablet 516b1021c8514f28b815929750f049da
I20260928 08:48:46.250428 18472 tablet_service.cc:1514] Processing DeleteTablet for tablet 516b1021c8514f28b815929750f049da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:48:46 UTC) from {username='slave'} at 127.0.0.1:50394
I20260928 08:48:46.250475 18342 tablet_service.cc:1514] Processing DeleteTablet for tablet 516b1021c8514f28b815929750f049da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:48:46 UTC) from {username='slave'} at 127.0.0.1:54432
I20260928 08:48:46.250788 18211 tablet_service.cc:1514] Processing DeleteTablet for tablet 516b1021c8514f28b815929750f049da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:48:46 UTC) from {username='slave'} at 127.0.0.1:58142
I20260928 08:48:46.252346 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 18149
I20260928 08:48:46.263725 18584 tablet_replica.cc:330] T 516b1021c8514f28b815929750f049da P 3d1afab0c50946919bdaaef6e2c99df5: stopping tablet replica
I20260928 08:48:46.264410 18583 tablet_replica.cc:330] T 516b1021c8514f28b815929750f049da P 5746d762f1ba492490b915617bb8b5c7: stopping tablet replica
I20260928 08:48:46.264474 18585 tablet_replica.cc:330] T 516b1021c8514f28b815929750f049da P 37f212074e104290951fd91fa590aa75: stopping tablet replica
I20260928 08:48:46.264961 18583 raft_consensus.cc:2229] T 516b1021c8514f28b815929750f049da P 5746d762f1ba492490b915617bb8b5c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:48:46.265033 18585 raft_consensus.cc:2229] T 516b1021c8514f28b815929750f049da P 37f212074e104290951fd91fa590aa75 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:48:46.265537 18583 raft_consensus.cc:2258] T 516b1021c8514f28b815929750f049da P 5746d762f1ba492490b915617bb8b5c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:48:46.266098 18585 raft_consensus.cc:2258] T 516b1021c8514f28b815929750f049da P 37f212074e104290951fd91fa590aa75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:48:46.267064 18584 raft_consensus.cc:2229] T 516b1021c8514f28b815929750f049da P 3d1afab0c50946919bdaaef6e2c99df5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:48:46.267671 18584 raft_consensus.cc:2258] T 516b1021c8514f28b815929750f049da P 3d1afab0c50946919bdaaef6e2c99df5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:48:46.267794 18583 ts_tablet_manager.cc:1905] T 516b1021c8514f28b815929750f049da P 5746d762f1ba492490b915617bb8b5c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:48:46.269744 18584 ts_tablet_manager.cc:1905] T 516b1021c8514f28b815929750f049da P 3d1afab0c50946919bdaaef6e2c99df5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:48:46.269791 18585 ts_tablet_manager.cc:1905] T 516b1021c8514f28b815929750f049da P 37f212074e104290951fd91fa590aa75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:48:46.276484 18583 ts_tablet_manager.cc:1918] T 516b1021c8514f28b815929750f049da P 5746d762f1ba492490b915617bb8b5c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:48:46.276690 18583 log.cc:1196] T 516b1021c8514f28b815929750f049da P 5746d762f1ba492490b915617bb8b5c7: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-1/wal/wals/516b1021c8514f28b815929750f049da
I20260928 08:48:46.277242 18583 ts_tablet_manager.cc:1939] T 516b1021c8514f28b815929750f049da P 5746d762f1ba492490b915617bb8b5c7: Deleting consensus metadata
I20260928 08:48:46.277237 18585 ts_tablet_manager.cc:1918] T 516b1021c8514f28b815929750f049da P 37f212074e104290951fd91fa590aa75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:48:46.277427 18585 log.cc:1196] T 516b1021c8514f28b815929750f049da P 37f212074e104290951fd91fa590aa75: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-0/wal/wals/516b1021c8514f28b815929750f049da
I20260928 08:48:46.278028 18585 ts_tablet_manager.cc:1939] T 516b1021c8514f28b815929750f049da P 37f212074e104290951fd91fa590aa75: Deleting consensus metadata
I20260928 08:48:46.279295 18076 catalog_manager.cc:4841] TS 5746d762f1ba492490b915617bb8b5c7 (127.0.97.2:40241): tablet 516b1021c8514f28b815929750f049da (table test-table [id=44d8ac381ba64ecb8a84dd202d2a3e9c]) successfully deleted
I20260928 08:48:46.279682 18074 catalog_manager.cc:4841] TS 37f212074e104290951fd91fa590aa75 (127.0.97.1:45617): tablet 516b1021c8514f28b815929750f049da (table test-table [id=44d8ac381ba64ecb8a84dd202d2a3e9c]) successfully deleted
I20260928 08:48:46.281390 18584 ts_tablet_manager.cc:1918] T 516b1021c8514f28b815929750f049da P 3d1afab0c50946919bdaaef6e2c99df5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:48:46.281605 18584 log.cc:1196] T 516b1021c8514f28b815929750f049da P 3d1afab0c50946919bdaaef6e2c99df5: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790585022442112-388-0/minicluster-data/ts-2/wal/wals/516b1021c8514f28b815929750f049da
I20260928 08:48:46.282130 18584 ts_tablet_manager.cc:1939] T 516b1021c8514f28b815929750f049da P 3d1afab0c50946919bdaaef6e2c99df5: Deleting consensus metadata
I20260928 08:48:46.284035 18074 catalog_manager.cc:4841] TS 3d1afab0c50946919bdaaef6e2c99df5 (127.0.97.3:34491): tablet 516b1021c8514f28b815929750f049da (table test-table [id=44d8ac381ba64ecb8a84dd202d2a3e9c]) successfully deleted
I20260928 08:48:46.284574 18570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:46.254090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58162 (local address 127.0.97.1:45617)
0928 08:48:46.254307 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:46.254313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:46.276520 (+ 22207us) server_negotiation.cc:407] Connection header received
0928 08:48:46.276834 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:46.276840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:46.276958 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:46.277048 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:46.277830 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:46.279074 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:46.280905 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:46.281367 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:46.281668 (+ 301us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:46.283015 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:46.283024 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:46.283038 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:48:46.283662 (+ 624us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:46.283712 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:48:46.283762 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:46.283907 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:46.284057 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 08:48:46.284174 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 08:48:46.285236 18271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:48:46.383579 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 18149
I20260928 08:48:46.402719 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 18280
I20260928 08:48:46.410053 18552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:46.403685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54442 (local address 127.0.97.2:40241)
0928 08:48:46.403941 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:46.403946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:46.403986 (+ 40us) server_negotiation.cc:407] Connection header received
0928 08:48:46.404045 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:46.404049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:46.404131 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:46.404211 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:46.404613 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:46.405428 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:46.406755 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:46.407188 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:46.407286 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:46.408559 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:46.408568 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:46.408584 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:48:46.409266 (+ 682us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:46.409317 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:48:46.409338 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:46.409430 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:46.409568 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 08:48:46.409661 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 08:48:46.410813 18402 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:48:46.501801 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 18280
I20260928 08:48:46.519269 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 18410
I20260928 08:48:46.526414 18550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:46.520134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50400 (local address 127.0.97.3:34491)
0928 08:48:46.520349 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:46.520354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:46.520374 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:48:46.520561 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:46.520565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:46.520674 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:46.520766 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:46.521152 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:46.521901 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:46.523312 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:46.523717 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:46.523906 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:46.524986 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:46.524995 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:46.525010 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:48:46.525612 (+ 602us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:46.525664 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:48:46.525690 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:46.525789 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:46.525910 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 08:48:46.526007 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 08:48:46.527175 18532 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:48:46.617529 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 18410
I20260928 08:48:46.634661 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 18059
I20260928 08:48:46.635984 18120 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:48:46.747460 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 18059
2026-09-28T08:48:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3359 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:48:46 dist-test-slave-n6xs krb5kdc[18593](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:48:46 dist-test-slave-n6xs krb5kdc[18593](info): set up 2 sockets
Sep 28 08:48:46 dist-test-slave-n6xs krb5kdc[18593](info): commencing operation
krb5kdc: starting...
W20260928 08:48:47.863996 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:48:48 dist-test-slave-n6xs krb5kdc[18593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:48:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:48:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:48:48.127944 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45703
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:45703
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:48:48.475989 18606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:48:48.476316 18606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:48:48.476493 18606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:48:48.484122 18606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:48:48.484256 18606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:48:48.484364 18606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:48:48.484467 18606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:48:48.484542 18606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:48:48.495072 18606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:45703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:45703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:48:48.496645 18606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:48:48.498185 18606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:48:48.511523 18611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:48.511325 18614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:48.511482 18612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:48:48.512739 18606 server_base.cc:1034] running on GCE node
I20260928 08:48:48.513691 18606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:48:48.515280 18606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:48:48.516530 18606 hybrid_clock.cc:648] HybridClock initialized: now 1790585328516465 us; error 63 us; skew 500 ppm
I20260928 08:48:48.516919 18606 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:48:48 dist-test-slave-n6xs krb5kdc[18593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585328, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:48:48.525704 18606 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:48:48.527609 18606 webserver.cc:460] Webserver started at http://127.0.97.62:44413/ using document root <none> and password file <none>
I20260928 08:48:48.528487 18606 fs_manager.cc:359] Metadata directory not provided
I20260928 08:48:48.528649 18606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:48:48.528991 18606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:48:48.532349 18606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "0556d478348c4485937a3c554b5bf6e8"
format_stamp: "Formatted at 2026-09-28 08:48:48 on dist-test-slave-n6xs"
server_key: "cf1e1e61c25f09290c9fe2353c4b04b9"
server_key_iv: "c9223d5c1f1e56cf4fcdcd3ed71821d9"
server_key_version: "encryptionkey@0"
I20260928 08:48:48.533207 18606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "0556d478348c4485937a3c554b5bf6e8"
format_stamp: "Formatted at 2026-09-28 08:48:48 on dist-test-slave-n6xs"
server_key: "cf1e1e61c25f09290c9fe2353c4b04b9"
server_key_iv: "c9223d5c1f1e56cf4fcdcd3ed71821d9"
server_key_version: "encryptionkey@0"
I20260928 08:48:48.538998 18606 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 08:48:48.543007 18621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:48.544691 18606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:48:48.544914 18606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "0556d478348c4485937a3c554b5bf6e8"
format_stamp: "Formatted at 2026-09-28 08:48:48 on dist-test-slave-n6xs"
server_key: "cf1e1e61c25f09290c9fe2353c4b04b9"
server_key_iv: "c9223d5c1f1e56cf4fcdcd3ed71821d9"
server_key_version: "encryptionkey@0"
I20260928 08:48:48.545148 18606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:48:48.596374 18606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:48:48.602216 18606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:48:48.602560 18606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:48:48.623337 18606 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:45703
I20260928 08:48:48.623451 18672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:45703 every 8 connection(s)
I20260928 08:48:48.625351 18606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:48:48.627512 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 18606
I20260928 08:48:48.627668 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:48:48.628188 388 external_mini_cluster.cc:1428] Setting key e534344be875230326b5c81f16612e93
I20260928 08:48:48.630141 18673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 08:48:48 dist-test-slave-n6xs krb5kdc[18593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:48:48.645350 18673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0556d478348c4485937a3c554b5bf6e8: Bootstrap starting.
I20260928 08:48:48.650120 18673 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0556d478348c4485937a3c554b5bf6e8: Neither blocks nor log segments found. Creating new log.
I20260928 08:48:48.651681 18673 log.cc:824] T 00000000000000000000000000000000 P 0556d478348c4485937a3c554b5bf6e8: Log is configured to *not* fsync() on all Append() calls
I20260928 08:48:48.655532 18673 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0556d478348c4485937a3c554b5bf6e8: No bootstrap required, opened a new log
I20260928 08:48:48.655997 18676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:48.630212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46018 (local address 127.0.97.62:45703)
0928 08:48:48.631098 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:48.631116 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:48.631173 (+ 57us) server_negotiation.cc:407] Connection header received
0928 08:48:48.633536 (+ 2363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:48.633570 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:48.634184 (+ 614us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:48.634708 (+ 524us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:48.636304 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:48.638499 (+ 2195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:48.640101 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:48.641280 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:48.641491 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:48.646140 (+ 4649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:48.646154 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:48.646179 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:48:48.646243 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:48.650669 (+ 4426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:48.651693 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:48.651706 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:48.651717 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:48.651983 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:48.652546 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:48.652551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:48.652555 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:48.653253 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:48.653558 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:48.654002 (+ 444us) server_negotiation.cc:299] Negotiation successful
0928 08:48:48.654506 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":352,"threads_started":1}
I20260928 08:48:48.663767 18673 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0556d478348c4485937a3c554b5bf6e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0556d478348c4485937a3c554b5bf6e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45703 } }
I20260928 08:48:48.664247 18673 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0556d478348c4485937a3c554b5bf6e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:48:48.664333 18673 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0556d478348c4485937a3c554b5bf6e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0556d478348c4485937a3c554b5bf6e8, State: Initialized, Role: FOLLOWER
I20260928 08:48:48.664880 18673 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0556d478348c4485937a3c554b5bf6e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0556d478348c4485937a3c554b5bf6e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45703 } }
I20260928 08:48:48.665096 18673 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0556d478348c4485937a3c554b5bf6e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:48:48.665205 18673 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0556d478348c4485937a3c554b5bf6e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:48:48.665346 18673 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0556d478348c4485937a3c554b5bf6e8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:48:48.667658 18673 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0556d478348c4485937a3c554b5bf6e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0556d478348c4485937a3c554b5bf6e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45703 } }
I20260928 08:48:48.668362 18673 leader_election.cc:302] T 00000000000000000000000000000000 P 0556d478348c4485937a3c554b5bf6e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0556d478348c4485937a3c554b5bf6e8; no voters:
I20260928 08:48:48.669060 18673 leader_election.cc:288] T 00000000000000000000000000000000 P 0556d478348c4485937a3c554b5bf6e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:48:48.669205 18678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0556d478348c4485937a3c554b5bf6e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:48:48.670007 18678 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0556d478348c4485937a3c554b5bf6e8 [term 1 LEADER]: Becoming Leader. State: Replica: 0556d478348c4485937a3c554b5bf6e8, State: Running, Role: LEADER
I20260928 08:48:48.670686 18678 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0556d478348c4485937a3c554b5bf6e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0556d478348c4485937a3c554b5bf6e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45703 } }
I20260928 08:48:48.671490 18673 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0556d478348c4485937a3c554b5bf6e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:48:48.673753 18680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0556d478348c4485937a3c554b5bf6e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0556d478348c4485937a3c554b5bf6e8. Latest consensus state: current_term: 1 leader_uuid: "0556d478348c4485937a3c554b5bf6e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0556d478348c4485937a3c554b5bf6e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45703 } } }
I20260928 08:48:48.674518 18679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0556d478348c4485937a3c554b5bf6e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0556d478348c4485937a3c554b5bf6e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0556d478348c4485937a3c554b5bf6e8" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 45703 } } }
I20260928 08:48:48.674823 18679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0556d478348c4485937a3c554b5bf6e8 [sys.catalog]: This master's current role is: LEADER
I20260928 08:48:48.674336 18680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0556d478348c4485937a3c554b5bf6e8 [sys.catalog]: This master's current role is: LEADER
I20260928 08:48:48.680558 18684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:48:48.686308 18684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:48:48.697000 18684 catalog_manager.cc:1347] Generated new cluster ID: 6759e05f58cb41c7b9df11b4e2002c3a
I20260928 08:48:48.697141 18684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:48:48.708366 18684 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:48:48.710485 18684 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:48:48.722846 18684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0556d478348c4485937a3c554b5bf6e8: Generated new TSK 0
I20260928 08:48:48.723925 18684 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:48:48.792514 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45703
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:48:49.158519 18699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:48:49.158890 18699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:48:49.159070 18699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:48:49.166862 18699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:48:49.167083 18699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:48:49.167270 18699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:48:49.177868 18699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:48:49.179364 18699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:48:49.180799 18699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:48:49.195710 18707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:49.196128 18704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:49.196408 18705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:48:49.197449 18699 server_base.cc:1034] running on GCE node
I20260928 08:48:49.198182 18699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:48:49.199445 18699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:48:49.200687 18699 hybrid_clock.cc:648] HybridClock initialized: now 1790585329200637 us; error 57 us; skew 500 ppm
I20260928 08:48:49.201074 18699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:48:49 dist-test-slave-n6xs krb5kdc[18593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:48:49.208329 18699 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:48:49.210165 18699 webserver.cc:460] Webserver started at http://127.0.97.1:44601/ using document root <none> and password file <none>
I20260928 08:48:49.210947 18699 fs_manager.cc:359] Metadata directory not provided
I20260928 08:48:49.211057 18699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:48:49.211336 18699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:48:49.214496 18699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-0/data/instance:
uuid: "bd674ba7ab0744e3a2b80346b1d5d175"
format_stamp: "Formatted at 2026-09-28 08:48:49 on dist-test-slave-n6xs"
server_key: "17e9e6bf7b58fb2824055787e51453cd"
server_key_iv: "0ef9eb234d3b1420db0dfa38db9e2645"
server_key_version: "encryptionkey@0"
I20260928 08:48:49.215293 18699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-0/wal/instance:
uuid: "bd674ba7ab0744e3a2b80346b1d5d175"
format_stamp: "Formatted at 2026-09-28 08:48:49 on dist-test-slave-n6xs"
server_key: "17e9e6bf7b58fb2824055787e51453cd"
server_key_iv: "0ef9eb234d3b1420db0dfa38db9e2645"
server_key_version: "encryptionkey@0"
I20260928 08:48:49.221127 18699 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 08:48:49.225319 18714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:49.226871 18699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:48:49.227087 18699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-0/wal
uuid: "bd674ba7ab0744e3a2b80346b1d5d175"
format_stamp: "Formatted at 2026-09-28 08:48:49 on dist-test-slave-n6xs"
server_key: "17e9e6bf7b58fb2824055787e51453cd"
server_key_iv: "0ef9eb234d3b1420db0dfa38db9e2645"
server_key_version: "encryptionkey@0"
I20260928 08:48:49.227326 18699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:48:49.260761 18699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:48:49.265734 18699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:48:49.266108 18699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:48:49.267233 18699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:48:49.268838 18699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:48:49.268918 18699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:49.269093 18699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:48:49.269158 18699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:49.305119 18699 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:43571
I20260928 08:48:49.305217 18826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:43571 every 8 connection(s)
I20260928 08:48:49.306675 18699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:48:49.313283 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 18699
I20260928 08:48:49.313489 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:48:49.313954 388 external_mini_cluster.cc:1428] Setting key 3dc3cc955172d1020e2f7dadcf3e79e7
Sep 28 08:48:49 dist-test-slave-n6xs krb5kdc[18593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:48:49.330508 18829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:49.309428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:56005 (local address 127.0.97.62:45703)
0928 08:48:49.309932 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:49.309937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:49.310836 (+ 899us) server_negotiation.cc:407] Connection header received
0928 08:48:49.312872 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:49.312879 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:49.312980 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:49.313110 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:49.314899 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:49.315796 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:49.318410 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:49.318830 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:49.318934 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:49.323608 (+ 4674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:49.323615 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:49.323618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:48:49.323658 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:49.326675 (+ 3017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:49.327608 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:49.327617 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:49.327621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:49.327784 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:49.328384 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:49.328390 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:49.328393 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:49.328746 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:49.328915 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:49.329776 (+ 861us) server_negotiation.cc:299] Negotiation successful
0928 08:48:49.330049 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":270,"threads_started":1}
I20260928 08:48:49.331972 18830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:49.309886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45703 (local address 127.0.97.1:56005)
0928 08:48:49.310631 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:49.310687 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:49.312557 (+ 1870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:49.313369 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:49.313378 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:49.314026 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:48:49.314694 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:49.314721 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:49.316024 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:49.316030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:49.318242 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:49.318262 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:49.319097 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:49.319107 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:49.319438 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:49.320274 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:48:49.320320 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:48:49.323297 (+ 2977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:48:49.326877 (+ 3580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:48:49.326888 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:48:49.326910 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:48:49.327449 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:48:49.327977 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:48:49.327984 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:48:49.327988 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:48:49.328235 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:48:49.328904 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:48:49.328913 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:48:49.329338 (+ 425us) client_negotiation.cc:769] Sending connection context
0928 08:48:49.329670 (+ 332us) client_negotiation.cc:240] Negotiation successful
0928 08:48:49.330307 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":289,"threads_started":1}
I20260928 08:48:49.333945 18827 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45703
I20260928 08:48:49.334317 18827 heartbeater.cc:464] Registering TS with master...
I20260928 08:48:49.335193 18827 heartbeater.cc:511] Master 127.0.97.62:45703 requested a full tablet report, sending...
I20260928 08:48:49.337739 18638 ts_manager.cc:194] Registered new tserver with Master: bd674ba7ab0744e3a2b80346b1d5d175 (127.0.97.1:43571)
I20260928 08:48:49.339385 18638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:56005
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 08:48:49.373333 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45703
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:48:49.728951 18833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:48:49.729357 18833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:48:49.729463 18833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:48:49.737466 18833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:48:49.737602 18833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:48:49.737722 18833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:48:49.748293 18833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:48:49.749696 18833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:48:49.751360 18833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:48:49.767133 18841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:49.767097 18839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:49.767086 18838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:48:49.773576 18833 server_base.cc:1034] running on GCE node
I20260928 08:48:49.774405 18833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:48:49.775768 18833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:48:49.777065 18833 hybrid_clock.cc:648] HybridClock initialized: now 1790585329776998 us; error 69 us; skew 500 ppm
I20260928 08:48:49.777451 18833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:48:49 dist-test-slave-n6xs krb5kdc[18593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:48:49.781064 18833 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:48:49.782922 18833 webserver.cc:460] Webserver started at http://127.0.97.2:41781/ using document root <none> and password file <none>
I20260928 08:48:49.783753 18833 fs_manager.cc:359] Metadata directory not provided
I20260928 08:48:49.783954 18833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:48:49.784260 18833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:48:49.787451 18833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-1/data/instance:
uuid: "d33377ab8a704fda9972ff32d6501742"
format_stamp: "Formatted at 2026-09-28 08:48:49 on dist-test-slave-n6xs"
server_key: "06b3ac72831e52961b960b049bf8ff2a"
server_key_iv: "0aaf13c0ca2d8065acc6a544670e79b2"
server_key_version: "encryptionkey@0"
I20260928 08:48:49.788482 18833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-1/wal/instance:
uuid: "d33377ab8a704fda9972ff32d6501742"
format_stamp: "Formatted at 2026-09-28 08:48:49 on dist-test-slave-n6xs"
server_key: "06b3ac72831e52961b960b049bf8ff2a"
server_key_iv: "0aaf13c0ca2d8065acc6a544670e79b2"
server_key_version: "encryptionkey@0"
I20260928 08:48:49.794494 18833 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260928 08:48:49.798688 18848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:49.800395 18833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:48:49.800632 18833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-1/wal
uuid: "d33377ab8a704fda9972ff32d6501742"
format_stamp: "Formatted at 2026-09-28 08:48:49 on dist-test-slave-n6xs"
server_key: "06b3ac72831e52961b960b049bf8ff2a"
server_key_iv: "0aaf13c0ca2d8065acc6a544670e79b2"
server_key_version: "encryptionkey@0"
I20260928 08:48:49.800925 18833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:48:49.822755 18833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:48:49.827692 18833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:48:49.828061 18833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:48:49.829202 18833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:48:49.830754 18833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:48:49.830876 18833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:49.831041 18833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:48:49.831112 18833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:49.867739 18833 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:35349
I20260928 08:48:49.867795 18960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:35349 every 8 connection(s)
I20260928 08:48:49.869208 18833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:48:49.869822 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 18833
I20260928 08:48:49.870026 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:48:49.870652 388 external_mini_cluster.cc:1428] Setting key 2c998658a93478bc31bc212eb1d2d500
Sep 28 08:48:49 dist-test-slave-n6xs krb5kdc[18593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:48:49.893564 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:49.872336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:48187 (local address 127.0.97.62:45703)
0928 08:48:49.872867 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:49.872873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:49.874033 (+ 1160us) server_negotiation.cc:407] Connection header received
0928 08:48:49.876200 (+ 2167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:49.876205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:49.876304 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:49.876440 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:49.878092 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:49.878949 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:49.880756 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:49.881244 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:49.881399 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:49.885862 (+ 4463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:49.885869 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:49.885873 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:48:49.885930 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:49.889214 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:49.890524 (+ 1310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:49.890530 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:49.890532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:49.890741 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:49.891431 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:49.891436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:49.891438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:49.891757 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:49.892021 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:49.893018 (+ 997us) server_negotiation.cc:299] Negotiation successful
0928 08:48:49.893281 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":337,"threads_started":1}
I20260928 08:48:49.895517 18964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:49.873048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45703 (local address 127.0.97.2:48187)
0928 08:48:49.873866 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:49.873913 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:49.875884 (+ 1971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:49.876710 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:49.876720 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:49.877280 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:48:49.877885 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:49.877905 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:49.879131 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:49.879135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:49.880575 (+ 1440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:49.880589 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:49.881504 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:49.881511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:49.881774 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:49.882539 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:48:49.882569 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:48:49.885471 (+ 2902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:48:49.889459 (+ 3988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:48:49.889474 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:48:49.889498 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:48:49.890308 (+ 810us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:48:49.890900 (+ 592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:48:49.890908 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:48:49.890915 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:48:49.891226 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:48:49.891970 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:48:49.891982 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:48:49.892560 (+ 578us) client_negotiation.cc:769] Sending connection context
0928 08:48:49.892918 (+ 358us) client_negotiation.cc:240] Negotiation successful
0928 08:48:49.893653 (+ 735us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":331,"threads_started":1}
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
I20260928 08:48:49.897495 18961 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45703
I20260928 08:48:49.897850 18961 heartbeater.cc:464] Registering TS with master...
I20260928 08:48:49.898768 18961 heartbeater.cc:511] Master 127.0.97.62:45703 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
I20260928 08:48:49.901024 18638 ts_manager.cc:194] Registered new tserver with Master: d33377ab8a704fda9972ff32d6501742 (127.0.97.2:35349)
I20260928 08:48:49.901939 18638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:48187
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:48:49.931775 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45703
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:48:50.285219 18967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:48:50.285578 18967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:48:50.285742 18967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:48:50.294543 18967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:48:50.294679 18967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:48:50.294878 18967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:48:50.306730 18967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:45703
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:48:50.308490 18967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:48:50.310397 18967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:48:50.325802 18975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:50.325979 18972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:50.325874 18973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:48:50.327569 18967 server_base.cc:1034] running on GCE node
I20260928 08:48:50.328393 18967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:48:50.329564 18967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:48:50.330825 18967 hybrid_clock.cc:648] HybridClock initialized: now 1790585330330752 us; error 87 us; skew 500 ppm
I20260928 08:48:50.331195 18967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:48:50 dist-test-slave-n6xs krb5kdc[18593](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:48:50.335069 18967 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:48:50.337019 18967 webserver.cc:460] Webserver started at http://127.0.97.3:43137/ using document root <none> and password file <none>
I20260928 08:48:50.337838 18967 fs_manager.cc:359] Metadata directory not provided
I20260928 08:48:50.338004 18967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:48:50.338354 18967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:48:50.341702 18967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-2/data/instance:
uuid: "909fc8ac33b44c1c910d6c940f44f300"
format_stamp: "Formatted at 2026-09-28 08:48:50 on dist-test-slave-n6xs"
server_key: "696c2d75eda46dfc3627173195f386a0"
server_key_iv: "893596341ca4d61764501a2ab8028c89"
server_key_version: "encryptionkey@0"
I20260928 08:48:50.342682 18967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-2/wal/instance:
uuid: "909fc8ac33b44c1c910d6c940f44f300"
format_stamp: "Formatted at 2026-09-28 08:48:50 on dist-test-slave-n6xs"
server_key: "696c2d75eda46dfc3627173195f386a0"
server_key_iv: "893596341ca4d61764501a2ab8028c89"
server_key_version: "encryptionkey@0"
I20260928 08:48:50.343338 18827 heartbeater.cc:503] Master 127.0.97.62:45703 was elected leader, sending a full tablet report...
I20260928 08:48:50.349866 18967 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260928 08:48:50.354413 18982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:50.356243 18967 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 08:48:50.356428 18967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-2/wal
uuid: "909fc8ac33b44c1c910d6c940f44f300"
format_stamp: "Formatted at 2026-09-28 08:48:50 on dist-test-slave-n6xs"
server_key: "696c2d75eda46dfc3627173195f386a0"
server_key_iv: "893596341ca4d61764501a2ab8028c89"
server_key_version: "encryptionkey@0"
I20260928 08:48:50.356621 18967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:48:50.372593 18967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:48:50.377668 18967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:48:50.377995 18967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:48:50.379139 18967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:48:50.380642 18967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:48:50.380722 18967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:50.380820 18967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:48:50.380877 18967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:50.417353 18967 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:33249
I20260928 08:48:50.417394 19094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:33249 every 8 connection(s)
I20260928 08:48:50.418817 18967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:48:50.428965 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 18967
I20260928 08:48:50.429174 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:48:50.429726 388 external_mini_cluster.cc:1428] Setting key 4346075fc78e47d61c0d3d1bbfd9ac8a
Sep 28 08:48:50 dist-test-slave-n6xs krb5kdc[18593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:48:50.441792 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:50.421840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:43521 (local address 127.0.97.62:45703)
0928 08:48:50.422423 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:50.422428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:50.423423 (+ 995us) server_negotiation.cc:407] Connection header received
0928 08:48:50.425538 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:50.425544 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:50.425647 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:50.425788 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:50.427535 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:50.428433 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:50.430954 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:50.431512 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:50.431621 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:50.435619 (+ 3998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:50.435625 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:50.435628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:48:50.435677 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:50.438568 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:50.439394 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:50.439398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:50.439401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:50.439532 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:50.440027 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:50.440031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:50.440033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:50.440341 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:50.440461 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:50.441280 (+ 819us) server_negotiation.cc:299] Negotiation successful
0928 08:48:50.441514 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":347,"threads_started":1}
I20260928 08:48:50.442991 19098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:50.422451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:45703 (local address 127.0.97.3:43521)
0928 08:48:50.423229 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:50.423274 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:50.425211 (+ 1937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:50.426081 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:50.426091 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:50.426678 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:48:50.427316 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:50.427337 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:50.428708 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:50.428716 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:50.430762 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:50.430782 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:50.431828 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:50.431841 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:50.432183 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:50.433022 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:48:50.433047 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:48:50.435378 (+ 2331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:48:50.438730 (+ 3352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:48:50.438738 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:48:50.438754 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:48:50.439250 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:48:50.439638 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:48:50.439642 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:48:50.439644 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:48:50.439917 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:48:50.440484 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:48:50.440492 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:48:50.440875 (+ 383us) client_negotiation.cc:769] Sending connection context
0928 08:48:50.441149 (+ 274us) client_negotiation.cc:240] Negotiation successful
0928 08:48:50.441657 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":306,"threads_started":1}
I20260928 08:48:50.444993 19095 heartbeater.cc:347] Connected to a master server at 127.0.97.62:45703
I20260928 08:48:50.445500 19095 heartbeater.cc:464] Registering TS with master...
I20260928 08:48:50.446508 19095 heartbeater.cc:511] Master 127.0.97.62:45703 requested a full tablet report, sending...
I20260928 08:48:50.448262 18637 ts_manager.cc:194] Registered new tserver with Master: 909fc8ac33b44c1c910d6c940f44f300 (127.0.97.3:33249)
I20260928 08:48:50.449266 18637 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:43521
I20260928 08:48:50.458909 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:48:50.478425 19097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:50.463998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46020 (local address 127.0.97.62:45703)
0928 08:48:50.464248 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:50.464257 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:50.464569 (+ 312us) server_negotiation.cc:407] Connection header received
0928 08:48:50.464739 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:50.464743 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:50.464811 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:50.464899 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:50.466475 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:50.467243 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:50.468673 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:50.469087 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:50.469191 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:50.472157 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:50.472163 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:50.472166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:48:50.472221 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:50.475197 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:50.476234 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:50.476240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:50.476242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:50.476373 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:50.476780 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:50.476784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:50.476785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:50.477119 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:50.477387 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:50.477782 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 08:48:50.478002 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 08:48:50.483184 18637 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:48:50.490705 18637 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:48:50.533792 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:50.508040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47210 (local address 127.0.97.3:33249)
0928 08:48:50.509021 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:50.509034 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:50.523054 (+ 14020us) server_negotiation.cc:407] Connection header received
0928 08:48:50.524681 (+ 1627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:50.524695 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:50.525094 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:50.525408 (+ 314us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:48:50.526497 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:50.527604 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:50.530723 (+ 3119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:50.532259 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:50.532481 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:50.532755 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:50.533163 (+ 408us) server_negotiation.cc:299] Negotiation successful
0928 08:48:50.533390 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":741,"thread_start_us":498,"threads_started":1}
I20260928 08:48:50.534955 19112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:50.507867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33249 (local address 127.0.0.1:47210)
0928 08:48:50.522790 (+ 14923us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:50.522874 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:50.523195 (+ 321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:50.525478 (+ 2283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:50.525491 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:50.525556 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:48:50.526222 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:50.526242 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:50.527784 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:50.527793 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:50.530485 (+ 2692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:50.530507 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:50.532527 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:50.532534 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:50.532824 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:50.532884 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 08:48:50.534246 (+ 1362us) client_negotiation.cc:240] Negotiation successful
0928 08:48:50.534503 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14511,"spinlock_wait_cycles":18688,"thread_start_us":15634,"threads_started":1}
I20260928 08:48:50.537595 19108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:50.508640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43571 (local address 127.0.0.1:41088)
0928 08:48:50.522830 (+ 14190us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:50.522876 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:50.523196 (+ 320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:50.530114 (+ 6918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:50.530124 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:50.530157 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:48:50.530647 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:50.530660 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:50.532647 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:50.532653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:50.534789 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:50.534803 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:50.536577 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:50.536584 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:50.536774 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:50.536817 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 08:48:50.536894 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 08:48:50.537037 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13976,"spinlock_wait_cycles":12672,"thread_start_us":19402,"threads_started":1}
I20260928 08:48:50.538280 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:50.510100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:35349 (local address 127.0.0.1:49590)
0928 08:48:50.528363 (+ 18263us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:50.528382 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:50.528568 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:50.529641 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:50.529649 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:50.529683 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:48:50.530199 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:50.530229 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:50.533053 (+ 2824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:50.533060 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:50.535387 (+ 2327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:50.535401 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:50.537551 (+ 2150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:50.537559 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:50.537794 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:50.537828 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 08:48:50.537923 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 08:48:50.538056 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18093,"thread_start_us":13388,"threads_started":1}
I20260928 08:48:50.538803 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:50.508313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41088 (local address 127.0.97.1:43571)
0928 08:48:50.529218 (+ 20905us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:50.529228 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:50.529260 (+ 32us) server_negotiation.cc:407] Connection header received
0928 08:48:50.529369 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:50.529377 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:50.529618 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:50.529849 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:48:50.530866 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:50.532471 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:50.534982 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:50.536346 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:50.537966 (+ 1620us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:50.538227 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:50.538306 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 08:48:50.538477 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20706,"thread_start_us":21270,"threads_started":1}
I20260928 08:48:50.539000 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:50.510445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49590 (local address 127.0.97.2:35349)
0928 08:48:50.525770 (+ 15325us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:50.525777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:50.528526 (+ 2749us) server_negotiation.cc:407] Connection header received
0928 08:48:50.528783 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:50.528796 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:50.529122 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:50.529364 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:48:50.530430 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:50.532893 (+ 2463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:50.535570 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:50.537023 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:50.537154 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:50.537376 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:50.538546 (+ 1170us) server_negotiation.cc:299] Negotiation successful
0928 08:48:50.538694 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15087,"thread_start_us":15299,"threads_started":1}
I20260928 08:48:50.540271 19030 tablet_service.cc:1467] Processing CreateTablet for tablet 903d387c491943a88f8b861585c84264 (DEFAULT_TABLE table=test-table [id=aaa8439f16fe4833ae98c048745b5f69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:48:50.542940 18762 tablet_service.cc:1467] Processing CreateTablet for tablet 903d387c491943a88f8b861585c84264 (DEFAULT_TABLE table=test-table [id=aaa8439f16fe4833ae98c048745b5f69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:48:50.543242 18896 tablet_service.cc:1467] Processing CreateTablet for tablet 903d387c491943a88f8b861585c84264 (DEFAULT_TABLE table=test-table [id=aaa8439f16fe4833ae98c048745b5f69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:48:50.544333 18762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 903d387c491943a88f8b861585c84264. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:48:50.544531 18896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 903d387c491943a88f8b861585c84264. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:48:50.547610 19030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 903d387c491943a88f8b861585c84264. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:48:50.557081 19114 tablet_bootstrap.cc:492] T 903d387c491943a88f8b861585c84264 P bd674ba7ab0744e3a2b80346b1d5d175: Bootstrap starting.
I20260928 08:48:50.561201 19114 tablet_bootstrap.cc:654] T 903d387c491943a88f8b861585c84264 P bd674ba7ab0744e3a2b80346b1d5d175: Neither blocks nor log segments found. Creating new log.
I20260928 08:48:50.563302 19114 log.cc:824] T 903d387c491943a88f8b861585c84264 P bd674ba7ab0744e3a2b80346b1d5d175: Log is configured to *not* fsync() on all Append() calls
I20260928 08:48:50.566598 19114 tablet_bootstrap.cc:492] T 903d387c491943a88f8b861585c84264 P bd674ba7ab0744e3a2b80346b1d5d175: No bootstrap required, opened a new log
I20260928 08:48:50.566921 19114 ts_tablet_manager.cc:1397] T 903d387c491943a88f8b861585c84264 P bd674ba7ab0744e3a2b80346b1d5d175: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.002s
I20260928 08:48:50.568295 19115 tablet_bootstrap.cc:492] T 903d387c491943a88f8b861585c84264 P d33377ab8a704fda9972ff32d6501742: Bootstrap starting.
I20260928 08:48:50.571403 19117 tablet_bootstrap.cc:492] T 903d387c491943a88f8b861585c84264 P 909fc8ac33b44c1c910d6c940f44f300: Bootstrap starting.
I20260928 08:48:50.572995 19115 tablet_bootstrap.cc:654] T 903d387c491943a88f8b861585c84264 P d33377ab8a704fda9972ff32d6501742: Neither blocks nor log segments found. Creating new log.
I20260928 08:48:50.574550 19115 log.cc:824] T 903d387c491943a88f8b861585c84264 P d33377ab8a704fda9972ff32d6501742: Log is configured to *not* fsync() on all Append() calls
I20260928 08:48:50.576486 19117 tablet_bootstrap.cc:654] T 903d387c491943a88f8b861585c84264 P 909fc8ac33b44c1c910d6c940f44f300: Neither blocks nor log segments found. Creating new log.
I20260928 08:48:50.576462 19114 raft_consensus.cc:348] T 903d387c491943a88f8b861585c84264 P bd674ba7ab0744e3a2b80346b1d5d175 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909fc8ac33b44c1c910d6c940f44f300" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33249 } } peers { permanent_uuid: "bd674ba7ab0744e3a2b80346b1d5d175" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43571 } } peers { permanent_uuid: "d33377ab8a704fda9972ff32d6501742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35349 } }
I20260928 08:48:50.576936 19114 raft_consensus.cc:374] T 903d387c491943a88f8b861585c84264 P bd674ba7ab0744e3a2b80346b1d5d175 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:48:50.577044 19114 raft_consensus.cc:729] T 903d387c491943a88f8b861585c84264 P bd674ba7ab0744e3a2b80346b1d5d175 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd674ba7ab0744e3a2b80346b1d5d175, State: Initialized, Role: FOLLOWER
I20260928 08:48:50.577867 19117 log.cc:824] T 903d387c491943a88f8b861585c84264 P 909fc8ac33b44c1c910d6c940f44f300: Log is configured to *not* fsync() on all Append() calls
I20260928 08:48:50.577792 19114 consensus_queue.cc:260] T 903d387c491943a88f8b861585c84264 P bd674ba7ab0744e3a2b80346b1d5d175 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909fc8ac33b44c1c910d6c940f44f300" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33249 } } peers { permanent_uuid: "bd674ba7ab0744e3a2b80346b1d5d175" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43571 } } peers { permanent_uuid: "d33377ab8a704fda9972ff32d6501742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35349 } }
I20260928 08:48:50.578485 19115 tablet_bootstrap.cc:492] T 903d387c491943a88f8b861585c84264 P d33377ab8a704fda9972ff32d6501742: No bootstrap required, opened a new log
I20260928 08:48:50.578923 19115 ts_tablet_manager.cc:1397] T 903d387c491943a88f8b861585c84264 P d33377ab8a704fda9972ff32d6501742: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20260928 08:48:50.580359 19114 ts_tablet_manager.cc:1428] T 903d387c491943a88f8b861585c84264 P bd674ba7ab0744e3a2b80346b1d5d175: Time spent starting tablet: real 0.013s user 0.012s sys 0.003s
I20260928 08:48:50.582038 19117 tablet_bootstrap.cc:492] T 903d387c491943a88f8b861585c84264 P 909fc8ac33b44c1c910d6c940f44f300: No bootstrap required, opened a new log
I20260928 08:48:50.582361 19117 ts_tablet_manager.cc:1397] T 903d387c491943a88f8b861585c84264 P 909fc8ac33b44c1c910d6c940f44f300: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260928 08:48:50.588681 19115 raft_consensus.cc:348] T 903d387c491943a88f8b861585c84264 P d33377ab8a704fda9972ff32d6501742 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909fc8ac33b44c1c910d6c940f44f300" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33249 } } peers { permanent_uuid: "bd674ba7ab0744e3a2b80346b1d5d175" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43571 } } peers { permanent_uuid: "d33377ab8a704fda9972ff32d6501742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35349 } }
I20260928 08:48:50.589102 19115 raft_consensus.cc:374] T 903d387c491943a88f8b861585c84264 P d33377ab8a704fda9972ff32d6501742 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:48:50.589274 19115 raft_consensus.cc:729] T 903d387c491943a88f8b861585c84264 P d33377ab8a704fda9972ff32d6501742 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d33377ab8a704fda9972ff32d6501742, State: Initialized, Role: FOLLOWER
I20260928 08:48:50.589902 19115 consensus_queue.cc:260] T 903d387c491943a88f8b861585c84264 P d33377ab8a704fda9972ff32d6501742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909fc8ac33b44c1c910d6c940f44f300" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33249 } } peers { permanent_uuid: "bd674ba7ab0744e3a2b80346b1d5d175" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43571 } } peers { permanent_uuid: "d33377ab8a704fda9972ff32d6501742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35349 } }
I20260928 08:48:50.591208 18961 heartbeater.cc:503] Master 127.0.97.62:45703 was elected leader, sending a full tablet report...
I20260928 08:48:50.591969 19115 ts_tablet_manager.cc:1428] T 903d387c491943a88f8b861585c84264 P d33377ab8a704fda9972ff32d6501742: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260928 08:48:50.591707 19117 raft_consensus.cc:348] T 903d387c491943a88f8b861585c84264 P 909fc8ac33b44c1c910d6c940f44f300 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909fc8ac33b44c1c910d6c940f44f300" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33249 } } peers { permanent_uuid: "bd674ba7ab0744e3a2b80346b1d5d175" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43571 } } peers { permanent_uuid: "d33377ab8a704fda9972ff32d6501742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35349 } }
I20260928 08:48:50.592257 19117 raft_consensus.cc:374] T 903d387c491943a88f8b861585c84264 P 909fc8ac33b44c1c910d6c940f44f300 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:48:50.592371 19117 raft_consensus.cc:729] T 903d387c491943a88f8b861585c84264 P 909fc8ac33b44c1c910d6c940f44f300 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 909fc8ac33b44c1c910d6c940f44f300, State: Initialized, Role: FOLLOWER
I20260928 08:48:50.592993 19117 consensus_queue.cc:260] T 903d387c491943a88f8b861585c84264 P 909fc8ac33b44c1c910d6c940f44f300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909fc8ac33b44c1c910d6c940f44f300" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33249 } } peers { permanent_uuid: "bd674ba7ab0744e3a2b80346b1d5d175" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43571 } } peers { permanent_uuid: "d33377ab8a704fda9972ff32d6501742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35349 } }
I20260928 08:48:50.594240 19095 heartbeater.cc:503] Master 127.0.97.62:45703 was elected leader, sending a full tablet report...
I20260928 08:48:50.594833 19117 ts_tablet_manager.cc:1428] T 903d387c491943a88f8b861585c84264 P 909fc8ac33b44c1c910d6c940f44f300: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
W20260928 08:48:50.621732 18962 tablet.cc:2369] T 903d387c491943a88f8b861585c84264 P d33377ab8a704fda9972ff32d6501742: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:48:50.670745 19096 tablet.cc:2369] T 903d387c491943a88f8b861585c84264 P 909fc8ac33b44c1c910d6c940f44f300: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:48:50.743927 19121 raft_consensus.cc:482] T 903d387c491943a88f8b861585c84264 P d33377ab8a704fda9972ff32d6501742 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:48:50.744266 19121 raft_consensus.cc:504] T 903d387c491943a88f8b861585c84264 P d33377ab8a704fda9972ff32d6501742 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909fc8ac33b44c1c910d6c940f44f300" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33249 } } peers { permanent_uuid: "bd674ba7ab0744e3a2b80346b1d5d175" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43571 } } peers { permanent_uuid: "d33377ab8a704fda9972ff32d6501742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35349 } }
I20260928 08:48:50.746181 19121 leader_election.cc:288] T 903d387c491943a88f8b861585c84264 P d33377ab8a704fda9972ff32d6501742 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 909fc8ac33b44c1c910d6c940f44f300 (127.0.97.3:33249), bd674ba7ab0744e3a2b80346b1d5d175 (127.0.97.1:43571)
I20260928 08:48:50.756313 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:50.746641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:49621 (local address 127.0.97.3:33249)
0928 08:48:50.746947 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:50.746957 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:50.747391 (+ 434us) server_negotiation.cc:407] Connection header received
0928 08:48:50.747613 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:50.747619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:50.747707 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:50.747841 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:48:50.748741 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:50.749938 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:50.753448 (+ 3510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:50.754773 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:50.754940 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:50.755059 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:50.755840 (+ 781us) server_negotiation.cc:299] Negotiation successful
0928 08:48:50.756014 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260928 08:48:50.756328 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:50.746633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:55771 (local address 127.0.97.1:43571)
0928 08:48:50.746868 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:50.746876 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:50.747480 (+ 604us) server_negotiation.cc:407] Connection header received
0928 08:48:50.747613 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:50.747619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:50.747722 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:50.747950 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:48:50.748794 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:50.749934 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:50.753450 (+ 3516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:50.754777 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:50.754940 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:50.755059 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:50.755803 (+ 744us) server_negotiation.cc:299] Negotiation successful
0928 08:48:50.756008 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 08:48:50.756569 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:50.746515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:43571 (local address 127.0.97.2:55771)
0928 08:48:50.747364 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:50.747379 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:50.747473 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:50.747885 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:50.747890 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:50.747989 (+ 99us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:48:50.748596 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:50.748608 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:50.750124 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:50.750130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:50.753178 (+ 3048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:50.753208 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:50.755174 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:50.755188 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:50.755501 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:50.755584 (+ 83us) client_negotiation.cc:769] Sending connection context
0928 08:48:50.755712 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 08:48:50.755957 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":724,"mutex_wait_us":373,"spinlock_wait_cycles":25344,"thread_start_us":463,"threads_started":1}
I20260928 08:48:50.756601 19123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:50.746515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:33249 (local address 127.0.97.2:49621)
0928 08:48:50.747252 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:50.747274 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:50.747461 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:50.747885 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:50.747890 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:50.747928 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:48:50.748507 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:50.748524 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:50.750124 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:50.750130 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:50.753176 (+ 3046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:50.753205 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:50.755174 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:50.755182 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:50.755496 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:50.755584 (+ 88us) client_negotiation.cc:769] Sending connection context
0928 08:48:50.755716 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 08:48:50.755957 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":349,"thread_start_us":442,"threads_started":1}
I20260928 08:48:50.757956 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903d387c491943a88f8b861585c84264" candidate_uuid: "d33377ab8a704fda9972ff32d6501742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909fc8ac33b44c1c910d6c940f44f300" is_pre_election: true
I20260928 08:48:50.758585 19050 raft_consensus.cc:2454] T 903d387c491943a88f8b861585c84264 P 909fc8ac33b44c1c910d6c940f44f300 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d33377ab8a704fda9972ff32d6501742 in term 0.
I20260928 08:48:50.758801 18782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903d387c491943a88f8b861585c84264" candidate_uuid: "d33377ab8a704fda9972ff32d6501742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd674ba7ab0744e3a2b80346b1d5d175" is_pre_election: true
I20260928 08:48:50.759503 18782 raft_consensus.cc:2454] T 903d387c491943a88f8b861585c84264 P bd674ba7ab0744e3a2b80346b1d5d175 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d33377ab8a704fda9972ff32d6501742 in term 0.
I20260928 08:48:50.759502 18849 leader_election.cc:302] T 903d387c491943a88f8b861585c84264 P d33377ab8a704fda9972ff32d6501742 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 909fc8ac33b44c1c910d6c940f44f300, d33377ab8a704fda9972ff32d6501742; no voters:
I20260928 08:48:50.760077 19121 raft_consensus.cc:2781] T 903d387c491943a88f8b861585c84264 P d33377ab8a704fda9972ff32d6501742 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:48:50.760290 19121 raft_consensus.cc:482] T 903d387c491943a88f8b861585c84264 P d33377ab8a704fda9972ff32d6501742 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:48:50.760437 19121 raft_consensus.cc:3037] T 903d387c491943a88f8b861585c84264 P d33377ab8a704fda9972ff32d6501742 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:48:50.763172 19121 raft_consensus.cc:504] T 903d387c491943a88f8b861585c84264 P d33377ab8a704fda9972ff32d6501742 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909fc8ac33b44c1c910d6c940f44f300" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33249 } } peers { permanent_uuid: "bd674ba7ab0744e3a2b80346b1d5d175" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43571 } } peers { permanent_uuid: "d33377ab8a704fda9972ff32d6501742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35349 } }
I20260928 08:48:50.763967 19121 leader_election.cc:288] T 903d387c491943a88f8b861585c84264 P d33377ab8a704fda9972ff32d6501742 [CANDIDATE]: Term 1 election: Requested vote from peers 909fc8ac33b44c1c910d6c940f44f300 (127.0.97.3:33249), bd674ba7ab0744e3a2b80346b1d5d175 (127.0.97.1:43571)
I20260928 08:48:50.764657 19050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903d387c491943a88f8b861585c84264" candidate_uuid: "d33377ab8a704fda9972ff32d6501742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "909fc8ac33b44c1c910d6c940f44f300"
I20260928 08:48:50.764658 18782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "903d387c491943a88f8b861585c84264" candidate_uuid: "d33377ab8a704fda9972ff32d6501742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd674ba7ab0744e3a2b80346b1d5d175"
I20260928 08:48:50.764925 18782 raft_consensus.cc:3037] T 903d387c491943a88f8b861585c84264 P bd674ba7ab0744e3a2b80346b1d5d175 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:48:50.764925 19050 raft_consensus.cc:3037] T 903d387c491943a88f8b861585c84264 P 909fc8ac33b44c1c910d6c940f44f300 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:48:50.768546 19050 raft_consensus.cc:2454] T 903d387c491943a88f8b861585c84264 P 909fc8ac33b44c1c910d6c940f44f300 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d33377ab8a704fda9972ff32d6501742 in term 1.
I20260928 08:48:50.768546 18782 raft_consensus.cc:2454] T 903d387c491943a88f8b861585c84264 P bd674ba7ab0744e3a2b80346b1d5d175 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d33377ab8a704fda9972ff32d6501742 in term 1.
I20260928 08:48:50.769254 18852 leader_election.cc:302] T 903d387c491943a88f8b861585c84264 P d33377ab8a704fda9972ff32d6501742 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 909fc8ac33b44c1c910d6c940f44f300, bd674ba7ab0744e3a2b80346b1d5d175, d33377ab8a704fda9972ff32d6501742; no voters:
I20260928 08:48:50.769680 19121 raft_consensus.cc:2781] T 903d387c491943a88f8b861585c84264 P d33377ab8a704fda9972ff32d6501742 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:48:50.770171 19121 raft_consensus.cc:686] T 903d387c491943a88f8b861585c84264 P d33377ab8a704fda9972ff32d6501742 [term 1 LEADER]: Becoming Leader. State: Replica: d33377ab8a704fda9972ff32d6501742, State: Running, Role: LEADER
I20260928 08:48:50.770748 19121 consensus_queue.cc:237] T 903d387c491943a88f8b861585c84264 P d33377ab8a704fda9972ff32d6501742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909fc8ac33b44c1c910d6c940f44f300" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33249 } } peers { permanent_uuid: "bd674ba7ab0744e3a2b80346b1d5d175" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43571 } } peers { permanent_uuid: "d33377ab8a704fda9972ff32d6501742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35349 } }
I20260928 08:48:50.776623 18638 catalog_manager.cc:5495] T 903d387c491943a88f8b861585c84264 P d33377ab8a704fda9972ff32d6501742 reported cstate change: term changed from 0 to 1, leader changed from <none> to d33377ab8a704fda9972ff32d6501742 (127.0.97.2). New cstate: current_term: 1 leader_uuid: "d33377ab8a704fda9972ff32d6501742" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "909fc8ac33b44c1c910d6c940f44f300" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd674ba7ab0744e3a2b80346b1d5d175" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d33377ab8a704fda9972ff32d6501742" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 35349 } health_report { overall_health: HEALTHY } } }
W20260928 08:48:50.809553 18828 tablet.cc:2369] T 903d387c491943a88f8b861585c84264 P bd674ba7ab0744e3a2b80346b1d5d175: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:48:50.843451 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:50.837565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49598 (local address 127.0.97.2:35349)
0928 08:48:50.837780 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:50.837785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:50.837850 (+ 65us) server_negotiation.cc:407] Connection header received
0928 08:48:50.838045 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:50.838050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:50.838131 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:50.838274 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:48:50.838726 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:50.839476 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:50.841128 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:50.841553 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:50.841669 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:50.842298 (+ 629us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:48:50.842844 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:48:50.842994 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:50.843186 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 08:48:50.843231 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 08:48:50.853178 19050 raft_consensus.cc:1261] T 903d387c491943a88f8b861585c84264 P 909fc8ac33b44c1c910d6c940f44f300 [term 1 FOLLOWER]: Refusing update from remote peer d33377ab8a704fda9972ff32d6501742: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:48:50.853226 18782 raft_consensus.cc:1261] T 903d387c491943a88f8b861585c84264 P bd674ba7ab0744e3a2b80346b1d5d175 [term 1 FOLLOWER]: Refusing update from remote peer d33377ab8a704fda9972ff32d6501742: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:48:50.854503 19125 consensus_queue.cc:1035] T 903d387c491943a88f8b861585c84264 P d33377ab8a704fda9972ff32d6501742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "909fc8ac33b44c1c910d6c940f44f300" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 33249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:48:50.854919 19121 consensus_queue.cc:1035] T 903d387c491943a88f8b861585c84264 P d33377ab8a704fda9972ff32d6501742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd674ba7ab0744e3a2b80346b1d5d175" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 43571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:48:50.896294 18637 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:48:50.896649 18637 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:48:50.901608 18637 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0556d478348c4485937a3c554b5bf6e8: Sending DeleteTablet for 3 replicas of tablet 903d387c491943a88f8b861585c84264
I20260928 08:48:50.903210 18762 tablet_service.cc:1514] Processing DeleteTablet for tablet 903d387c491943a88f8b861585c84264 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:48:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:41088
I20260928 08:48:50.903208 19030 tablet_service.cc:1514] Processing DeleteTablet for tablet 903d387c491943a88f8b861585c84264 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:48:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:47210
I20260928 08:48:50.904093 18896 tablet_service.cc:1514] Processing DeleteTablet for tablet 903d387c491943a88f8b861585c84264 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:48:50 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:49590
I20260928 08:48:50.905278 19138 tablet_replica.cc:330] T 903d387c491943a88f8b861585c84264 P 909fc8ac33b44c1c910d6c940f44f300: stopping tablet replica
I20260928 08:48:50.905369 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 18699
I20260928 08:48:50.905436 19139 tablet_replica.cc:330] T 903d387c491943a88f8b861585c84264 P bd674ba7ab0744e3a2b80346b1d5d175: stopping tablet replica
I20260928 08:48:50.906611 19138 raft_consensus.cc:2229] T 903d387c491943a88f8b861585c84264 P 909fc8ac33b44c1c910d6c940f44f300 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:48:50.907351 19138 raft_consensus.cc:2258] T 903d387c491943a88f8b861585c84264 P 909fc8ac33b44c1c910d6c940f44f300 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:48:50.907925 19139 raft_consensus.cc:2229] T 903d387c491943a88f8b861585c84264 P bd674ba7ab0744e3a2b80346b1d5d175 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:48:50.908628 19139 raft_consensus.cc:2258] T 903d387c491943a88f8b861585c84264 P bd674ba7ab0744e3a2b80346b1d5d175 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:48:50.911638 19139 ts_tablet_manager.cc:1905] T 903d387c491943a88f8b861585c84264 P bd674ba7ab0744e3a2b80346b1d5d175: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:48:50.918339 19138 ts_tablet_manager.cc:1905] T 903d387c491943a88f8b861585c84264 P 909fc8ac33b44c1c910d6c940f44f300: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:48:50.922542 19140 tablet_replica.cc:330] T 903d387c491943a88f8b861585c84264 P d33377ab8a704fda9972ff32d6501742: stopping tablet replica
I20260928 08:48:50.923504 19140 raft_consensus.cc:2229] T 903d387c491943a88f8b861585c84264 P d33377ab8a704fda9972ff32d6501742 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:48:50.924401 19140 raft_consensus.cc:2258] T 903d387c491943a88f8b861585c84264 P d33377ab8a704fda9972ff32d6501742 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:48:50.927204 19140 ts_tablet_manager.cc:1905] T 903d387c491943a88f8b861585c84264 P d33377ab8a704fda9972ff32d6501742: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:48:50.928023 19138 ts_tablet_manager.cc:1918] T 903d387c491943a88f8b861585c84264 P 909fc8ac33b44c1c910d6c940f44f300: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:48:50.928220 19138 log.cc:1196] T 903d387c491943a88f8b861585c84264 P 909fc8ac33b44c1c910d6c940f44f300: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-2/wal/wals/903d387c491943a88f8b861585c84264
I20260928 08:48:50.928797 19138 ts_tablet_manager.cc:1939] T 903d387c491943a88f8b861585c84264 P 909fc8ac33b44c1c910d6c940f44f300: Deleting consensus metadata
I20260928 08:48:50.930963 18622 catalog_manager.cc:4841] TS 909fc8ac33b44c1c910d6c940f44f300 (127.0.97.3:33249): tablet 903d387c491943a88f8b861585c84264 (table test-table [id=aaa8439f16fe4833ae98c048745b5f69]) successfully deleted
Sep 28 08:48:50 dist-test-slave-n6xs krb5kdc[18593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:48:50.935693 19139 ts_tablet_manager.cc:1918] T 903d387c491943a88f8b861585c84264 P bd674ba7ab0744e3a2b80346b1d5d175: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:48:50.936084 19139 log.cc:1196] T 903d387c491943a88f8b861585c84264 P bd674ba7ab0744e3a2b80346b1d5d175: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-0/wal/wals/903d387c491943a88f8b861585c84264
I20260928 08:48:50.936553 19140 ts_tablet_manager.cc:1918] T 903d387c491943a88f8b861585c84264 P d33377ab8a704fda9972ff32d6501742: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:48:50.936822 19139 ts_tablet_manager.cc:1939] T 903d387c491943a88f8b861585c84264 P bd674ba7ab0744e3a2b80346b1d5d175: Deleting consensus metadata
I20260928 08:48:50.936817 19140 log.cc:1196] T 903d387c491943a88f8b861585c84264 P d33377ab8a704fda9972ff32d6501742: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790585022442112-388-0/minicluster-data/ts-1/wal/wals/903d387c491943a88f8b861585c84264
I20260928 08:48:50.937474 19140 ts_tablet_manager.cc:1939] T 903d387c491943a88f8b861585c84264 P d33377ab8a704fda9972ff32d6501742: Deleting consensus metadata
I20260928 08:48:50.939217 18625 catalog_manager.cc:4841] TS bd674ba7ab0744e3a2b80346b1d5d175 (127.0.97.1:43571): tablet 903d387c491943a88f8b861585c84264 (table test-table [id=aaa8439f16fe4833ae98c048745b5f69]) successfully deleted
I20260928 08:48:50.939841 18623 catalog_manager.cc:4841] TS d33377ab8a704fda9972ff32d6501742 (127.0.97.2:35349): tablet 903d387c491943a88f8b861585c84264 (table test-table [id=aaa8439f16fe4833ae98c048745b5f69]) successfully deleted
I20260928 08:48:50.944010 19110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:50.908574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41094 (local address 127.0.97.1:43571)
0928 08:48:50.908765 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:50.908772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:50.922536 (+ 13764us) server_negotiation.cc:407] Connection header received
0928 08:48:50.922882 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:50.922889 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:50.922994 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:50.923069 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:50.925094 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:50.926304 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:50.928504 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:50.929209 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:50.929335 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:50.936520 (+ 7185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:50.936542 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:50.936565 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:48:50.936619 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:50.940781 (+ 4162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:50.941655 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:50.941663 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:50.941665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:50.941829 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:50.942275 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:50.942280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:50.942285 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:50.942676 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:50.943000 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:50.943355 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 08:48:50.943596 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 08:48:50.945017 18822 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:48:51.041393 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 18699
I20260928 08:48:51.060263 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 18833
Sep 28 08:48:51 dist-test-slave-n6xs krb5kdc[18593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 08:48:51.078840 19113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:51.061267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49600 (local address 127.0.97.2:35349)
0928 08:48:51.061470 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:51.061476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:51.061498 (+ 22us) server_negotiation.cc:407] Connection header received
0928 08:48:51.061682 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:51.061690 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:51.061781 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:51.061916 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:51.063226 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:51.064195 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:51.065589 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:51.066059 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:51.066221 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:51.071421 (+ 5200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:51.071451 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:51.071468 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:48:51.071519 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:51.075075 (+ 3556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:51.076298 (+ 1223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:51.076307 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:51.076310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:51.076479 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:51.076934 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:51.076937 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:51.076939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:51.077346 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:51.077611 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:51.078110 (+ 499us) server_negotiation.cc:299] Negotiation successful
0928 08:48:51.078420 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 08:48:51.079710 18956 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:48:51.199275 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 18833
I20260928 08:48:51.218904 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 18967
Sep 28 08:48:51 dist-test-slave-n6xs krb5kdc[18593](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 08:48:51.236236 19109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:51.219709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47224 (local address 127.0.97.3:33249)
0928 08:48:51.219982 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:51.219989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:51.220109 (+ 120us) server_negotiation.cc:407] Connection header received
0928 08:48:51.220382 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:51.220392 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:51.220463 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:51.220586 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:51.221989 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:51.222745 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:51.224166 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:51.224575 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:51.224743 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:51.229650 (+ 4907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:51.229669 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:51.229686 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:48:51.229731 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:51.232922 (+ 3191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:51.233908 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:51.233916 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:51.233918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:51.234072 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:51.234694 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:51.234697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:51.234700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:51.235083 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:51.235314 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:51.235674 (+ 360us) server_negotiation.cc:299] Negotiation successful
0928 08:48:51.235948 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260928 08:48:51.236990 19090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:48:51.328857 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 18967
I20260928 08:48:51.346300 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 18606
I20260928 08:48:51.347632 18668 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:48:51.465370 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 18606
2026-09-28T08:48:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4718 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T08:48:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:48:51Z Disabled control of system clock
I20260928 08:48:51.545766 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37465
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:35505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:37465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 08:48:51.905483 19149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:48:51.905980 19149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:48:51.906152 19149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:48:51.913975 19149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:48:51.914109 19149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:48:51.914196 19149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:48:51.914300 19149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:48:51.914391 19149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:48:51.924294 19149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:37465
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:37465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:48:51.925781 19149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:48:51.927785 19149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:48:51.941380 19154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:51.941380 19155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:51.941418 19157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:48:51.942421 19149 server_base.cc:1034] running on GCE node
I20260928 08:48:51.943408 19149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:48:51.945189 19149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:48:51.946429 19149 hybrid_clock.cc:648] HybridClock initialized: now 1790585331946380 us; error 46 us; skew 500 ppm
I20260928 08:48:51.946867 19149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:48:51.949762 19149 webserver.cc:460] Webserver started at http://127.0.97.62:33695/ using document root <none> and password file <none>
I20260928 08:48:51.950611 19149 fs_manager.cc:359] Metadata directory not provided
I20260928 08:48:51.950719 19149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:48:51.951076 19149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:48:51.954552 19149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "2bc99e5bd7004181bbcf8b3d6cc0a70e"
format_stamp: "Formatted at 2026-09-28 08:48:51 on dist-test-slave-n6xs"
server_key: "86a7155c85667f96f481a51c1f24ffad"
server_key_iv: "4512abf143e72e0c24adc0b93dae6070"
server_key_version: "encryptionkey@0"
I20260928 08:48:51.955543 19149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "2bc99e5bd7004181bbcf8b3d6cc0a70e"
format_stamp: "Formatted at 2026-09-28 08:48:51 on dist-test-slave-n6xs"
server_key: "86a7155c85667f96f481a51c1f24ffad"
server_key_iv: "4512abf143e72e0c24adc0b93dae6070"
server_key_version: "encryptionkey@0"
I20260928 08:48:51.961755 19149 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260928 08:48:51.966133 19163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:51.967983 19149 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 08:48:51.968226 19149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "2bc99e5bd7004181bbcf8b3d6cc0a70e"
format_stamp: "Formatted at 2026-09-28 08:48:51 on dist-test-slave-n6xs"
server_key: "86a7155c85667f96f481a51c1f24ffad"
server_key_iv: "4512abf143e72e0c24adc0b93dae6070"
server_key_version: "encryptionkey@0"
I20260928 08:48:51.968508 19149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:48:51.999046 19149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:48:52.000101 19149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:48:52.000447 19149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:48:52.022462 19149 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:37465
I20260928 08:48:52.022486 19214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:37465 every 8 connection(s)
I20260928 08:48:52.024096 19149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:48:52.025579 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 19149
I20260928 08:48:52.025774 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:48:52.026388 388 external_mini_cluster.cc:1428] Setting key ac8d3f76af4c55bcdeab8f36350ed587
I20260928 08:48:52.029583 19215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:48:52.041270 19218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:52.028544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35136 (local address 127.0.97.62:37465)
0928 08:48:52.029707 (+ 1163us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:52.029725 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:52.029788 (+ 63us) server_negotiation.cc:407] Connection header received
0928 08:48:52.031498 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:52.031523 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:52.032130 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:52.032601 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:52.033239 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:52.034628 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:52.036275 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:52.037022 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:52.037250 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:52.038558 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:52.038569 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:52.038587 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:48:52.039243 (+ 656us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:52.039341 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:48:52.039366 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:52.039515 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:52.039759 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 08:48:52.040195 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":401,"threads_started":1}
I20260928 08:48:52.041512 19215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bc99e5bd7004181bbcf8b3d6cc0a70e: Bootstrap starting.
I20260928 08:48:52.045074 19215 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bc99e5bd7004181bbcf8b3d6cc0a70e: Neither blocks nor log segments found. Creating new log.
I20260928 08:48:52.046218 19215 log.cc:824] T 00000000000000000000000000000000 P 2bc99e5bd7004181bbcf8b3d6cc0a70e: Log is configured to *not* fsync() on all Append() calls
I20260928 08:48:52.049352 19215 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bc99e5bd7004181bbcf8b3d6cc0a70e: No bootstrap required, opened a new log
I20260928 08:48:52.057106 19215 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bc99e5bd7004181bbcf8b3d6cc0a70e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc99e5bd7004181bbcf8b3d6cc0a70e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37465 } }
I20260928 08:48:52.057539 19215 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bc99e5bd7004181bbcf8b3d6cc0a70e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:48:52.057682 19215 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bc99e5bd7004181bbcf8b3d6cc0a70e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bc99e5bd7004181bbcf8b3d6cc0a70e, State: Initialized, Role: FOLLOWER
I20260928 08:48:52.058450 19215 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bc99e5bd7004181bbcf8b3d6cc0a70e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc99e5bd7004181bbcf8b3d6cc0a70e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37465 } }
I20260928 08:48:52.058735 19215 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bc99e5bd7004181bbcf8b3d6cc0a70e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:48:52.058890 19215 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bc99e5bd7004181bbcf8b3d6cc0a70e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:48:52.059084 19215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bc99e5bd7004181bbcf8b3d6cc0a70e [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:48:52.061563 19215 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bc99e5bd7004181bbcf8b3d6cc0a70e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc99e5bd7004181bbcf8b3d6cc0a70e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37465 } }
I20260928 08:48:52.062204 19215 leader_election.cc:302] T 00000000000000000000000000000000 P 2bc99e5bd7004181bbcf8b3d6cc0a70e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bc99e5bd7004181bbcf8b3d6cc0a70e; no voters:
I20260928 08:48:52.062870 19215 leader_election.cc:288] T 00000000000000000000000000000000 P 2bc99e5bd7004181bbcf8b3d6cc0a70e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:48:52.063088 19220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bc99e5bd7004181bbcf8b3d6cc0a70e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:48:52.063968 19220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bc99e5bd7004181bbcf8b3d6cc0a70e [term 1 LEADER]: Becoming Leader. State: Replica: 2bc99e5bd7004181bbcf8b3d6cc0a70e, State: Running, Role: LEADER
I20260928 08:48:52.064682 19220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bc99e5bd7004181bbcf8b3d6cc0a70e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc99e5bd7004181bbcf8b3d6cc0a70e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37465 } }
I20260928 08:48:52.065384 19215 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bc99e5bd7004181bbcf8b3d6cc0a70e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:48:52.067956 19221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bc99e5bd7004181bbcf8b3d6cc0a70e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bc99e5bd7004181bbcf8b3d6cc0a70e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc99e5bd7004181bbcf8b3d6cc0a70e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37465 } } }
I20260928 08:48:52.068513 19221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bc99e5bd7004181bbcf8b3d6cc0a70e [sys.catalog]: This master's current role is: LEADER
I20260928 08:48:52.069137 19222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bc99e5bd7004181bbcf8b3d6cc0a70e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bc99e5bd7004181bbcf8b3d6cc0a70e. Latest consensus state: current_term: 1 leader_uuid: "2bc99e5bd7004181bbcf8b3d6cc0a70e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bc99e5bd7004181bbcf8b3d6cc0a70e" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 37465 } } }
I20260928 08:48:52.069540 19222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bc99e5bd7004181bbcf8b3d6cc0a70e [sys.catalog]: This master's current role is: LEADER
I20260928 08:48:52.070418 19225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:48:52.076718 19225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:48:52.087270 19225 catalog_manager.cc:1347] Generated new cluster ID: 0cb9e1bd9e1c4d778a6a637509dca996
I20260928 08:48:52.087422 19225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:48:52.107342 19225 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:48:52.109376 19225 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:48:52.126144 19225 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bc99e5bd7004181bbcf8b3d6cc0a70e: Generated new TSK 0
I20260928 08:48:52.127442 19225 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:48:52.143036 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37465
--builtin_ntp_servers=127.0.97.20:35505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 08:48:52.512485 19239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:48:52.512812 19239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:48:52.512902 19239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:48:52.520641 19239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:48:52.520764 19239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:48:52.520874 19239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:48:52.531770 19239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37465
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:48:52.533207 19239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:48:52.535023 19239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:48:52.549904 19245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:52.550004 19247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:52.549962 19244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:48:52.550904 19239 server_base.cc:1034] running on GCE node
I20260928 08:48:52.551602 19239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:48:52.552759 19239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:48:52.554031 19239 hybrid_clock.cc:648] HybridClock initialized: now 1790585332553976 us; error 56 us; skew 500 ppm
I20260928 08:48:52.554468 19239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:48:52.557225 19239 webserver.cc:460] Webserver started at http://127.0.97.1:41733/ using document root <none> and password file <none>
I20260928 08:48:52.558185 19239 fs_manager.cc:359] Metadata directory not provided
I20260928 08:48:52.558333 19239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:48:52.558686 19239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:48:52.561923 19239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-0/data/instance:
uuid: "389a8c16d22641a4ba3f4247d5200499"
format_stamp: "Formatted at 2026-09-28 08:48:52 on dist-test-slave-n6xs"
server_key: "fafe255aa60aca533eb0c857cfba5280"
server_key_iv: "407ea3520fc7ee1da2ee2c33da8cee10"
server_key_version: "encryptionkey@0"
I20260928 08:48:52.562861 19239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-0/wal/instance:
uuid: "389a8c16d22641a4ba3f4247d5200499"
format_stamp: "Formatted at 2026-09-28 08:48:52 on dist-test-slave-n6xs"
server_key: "fafe255aa60aca533eb0c857cfba5280"
server_key_iv: "407ea3520fc7ee1da2ee2c33da8cee10"
server_key_version: "encryptionkey@0"
I20260928 08:48:52.569176 19239 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 08:48:52.573537 19253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:52.575186 19239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260928 08:48:52.575397 19239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-0/wal
uuid: "389a8c16d22641a4ba3f4247d5200499"
format_stamp: "Formatted at 2026-09-28 08:48:52 on dist-test-slave-n6xs"
server_key: "fafe255aa60aca533eb0c857cfba5280"
server_key_iv: "407ea3520fc7ee1da2ee2c33da8cee10"
server_key_version: "encryptionkey@0"
I20260928 08:48:52.575670 19239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:48:52.595873 19239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:48:52.596885 19239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:48:52.597230 19239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:48:52.598361 19239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:48:52.600078 19239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:48:52.600214 19239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:52.600366 19239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:48:52.600456 19239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:52.636638 19239 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:40741
I20260928 08:48:52.636705 19365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:40741 every 8 connection(s)
I20260928 08:48:52.638195 19239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:48:52.641417 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 19239
I20260928 08:48:52.641628 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:48:52.642091 388 external_mini_cluster.cc:1428] Setting key d0d40f708c20e079149ae27de59078aa
I20260928 08:48:52.645627 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37465
--builtin_ntp_servers=127.0.97.20:35505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 08:48:52.653934 19368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:52.641282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:35481 (local address 127.0.97.62:37465)
0928 08:48:52.641803 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:52.641810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:52.643407 (+ 1597us) server_negotiation.cc:407] Connection header received
0928 08:48:52.645842 (+ 2435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:52.645849 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:52.645960 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:52.646080 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:52.647279 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:52.648495 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:52.650326 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:52.651143 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:52.651310 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:52.652957 (+ 1647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:52.652966 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:52.652974 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:48:52.653005 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:52.653065 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:48:52.653085 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:52.653192 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:52.653518 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 08:48:52.653617 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":254,"threads_started":1}
I20260928 08:48:52.655616 19369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:52.641973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37465 (local address 127.0.97.1:35481)
0928 08:48:52.643215 (+ 1242us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:52.643256 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:52.645477 (+ 2221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:52.646312 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:52.646322 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:52.646431 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:48:52.647067 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:52.647085 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:52.648691 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:52.648695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:52.650091 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:52.650108 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:52.651410 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:52.651415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:52.651662 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:52.652554 (+ 892us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:48:52.652556 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:48:52.652605 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:48:52.652706 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:48:52.653224 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:48:52.653232 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:48:52.653239 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 08:48:52.653496 (+ 257us) client_negotiation.cc:240] Negotiation successful
0928 08:48:52.654027 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":380,"threads_started":1}
I20260928 08:48:52.657671 19366 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37465
I20260928 08:48:52.658073 19366 heartbeater.cc:464] Registering TS with master...
I20260928 08:48:52.658944 19366 heartbeater.cc:511] Master 127.0.97.62:37465 requested a full tablet report, sending...
I20260928 08:48:52.661527 19180 ts_manager.cc:194] Registered new tserver with Master: 389a8c16d22641a4ba3f4247d5200499 (127.0.97.1:40741)
I20260928 08:48:52.663130 19180 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:35481
W20260928 08:48:53.004063 19370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:48:53.004456 19370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:48:53.004630 19370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:48:53.013845 19370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:48:53.014014 19370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:48:53.014190 19370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:48:53.024040 19370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37465
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:48:53.025374 19370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:48:53.027297 19370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:48:53.041929 19375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:53.042037 19376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:53.041893 19378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:48:53.043120 19370 server_base.cc:1034] running on GCE node
I20260928 08:48:53.043901 19370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:48:53.044981 19370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:48:53.046254 19370 hybrid_clock.cc:648] HybridClock initialized: now 1790585333046203 us; error 46 us; skew 500 ppm
I20260928 08:48:53.046634 19370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:48:53.049352 19370 webserver.cc:460] Webserver started at http://127.0.97.2:44981/ using document root <none> and password file <none>
I20260928 08:48:53.050191 19370 fs_manager.cc:359] Metadata directory not provided
I20260928 08:48:53.050300 19370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:48:53.050642 19370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:48:53.053980 19370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-1/data/instance:
uuid: "b94a04c527ff40ff86ae8a15ad1fe82a"
format_stamp: "Formatted at 2026-09-28 08:48:53 on dist-test-slave-n6xs"
server_key: "e4f1fc6936f6eb9e8b716a861a665de6"
server_key_iv: "253b3722e89adab83e2399b4ad58a7ea"
server_key_version: "encryptionkey@0"
I20260928 08:48:53.054924 19370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-1/wal/instance:
uuid: "b94a04c527ff40ff86ae8a15ad1fe82a"
format_stamp: "Formatted at 2026-09-28 08:48:53 on dist-test-slave-n6xs"
server_key: "e4f1fc6936f6eb9e8b716a861a665de6"
server_key_iv: "253b3722e89adab83e2399b4ad58a7ea"
server_key_version: "encryptionkey@0"
I20260928 08:48:53.060947 19370 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 08:48:53.065161 19384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:53.067055 19370 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260928 08:48:53.067297 19370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-1/wal
uuid: "b94a04c527ff40ff86ae8a15ad1fe82a"
format_stamp: "Formatted at 2026-09-28 08:48:53 on dist-test-slave-n6xs"
server_key: "e4f1fc6936f6eb9e8b716a861a665de6"
server_key_iv: "253b3722e89adab83e2399b4ad58a7ea"
server_key_version: "encryptionkey@0"
I20260928 08:48:53.067551 19370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:48:53.082701 19370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:48:53.083653 19370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:48:53.084012 19370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:48:53.085105 19370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:48:53.086817 19370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:48:53.086949 19370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:53.087107 19370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:48:53.087200 19370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:53.122736 19370 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:34999
I20260928 08:48:53.122789 19496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:34999 every 8 connection(s)
I20260928 08:48:53.124291 19370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:48:53.124869 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 19370
I20260928 08:48:53.125136 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:48:53.125739 388 external_mini_cluster.cc:1428] Setting key cedbd6431cdcc1b4a15b40ac304c77cc
I20260928 08:48:53.129653 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37465
--builtin_ntp_servers=127.0.97.20:35505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 08:48:53.138830 19368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:53.127236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:44351 (local address 127.0.97.62:37465)
0928 08:48:53.127403 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:53.127407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:53.129375 (+ 1968us) server_negotiation.cc:407] Connection header received
0928 08:48:53.131431 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:53.131438 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:53.131554 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:53.131687 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:53.133012 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:53.133798 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:53.135535 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:53.136079 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:53.136230 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:53.137934 (+ 1704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:53.137941 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:53.137945 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:48:53.137977 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:53.138011 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:48:53.138024 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:53.138129 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:53.138380 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 08:48:53.138493 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 08:48:53.140208 19499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:53.128092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37465 (local address 127.0.97.2:44351)
0928 08:48:53.129195 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:53.129246 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:53.131097 (+ 1851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:53.132022 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:53.132032 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:53.132143 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:48:53.132797 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:53.132817 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:53.133991 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:53.133999 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:53.135359 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:53.135373 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:53.136340 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:53.136348 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:53.136618 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:53.137491 (+ 873us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:48:53.137496 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:48:53.137572 (+ 76us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:48:53.137694 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:48:53.138131 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:48:53.138146 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:48:53.138156 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 08:48:53.138366 (+ 210us) client_negotiation.cc:240] Negotiation successful
0928 08:48:53.138853 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":380,"threads_started":1}
I20260928 08:48:53.141979 19497 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37465
I20260928 08:48:53.142385 19497 heartbeater.cc:464] Registering TS with master...
I20260928 08:48:53.143484 19497 heartbeater.cc:511] Master 127.0.97.62:37465 requested a full tablet report, sending...
I20260928 08:48:53.145539 19180 ts_manager.cc:194] Registered new tserver with Master: b94a04c527ff40ff86ae8a15ad1fe82a (127.0.97.2:34999)
I20260928 08:48:53.146927 19180 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:44351
W20260928 08:48:53.485769 19500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:48:53.486116 19500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:48:53.486205 19500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:48:53.493983 19500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:48:53.494110 19500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:48:53.494220 19500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:48:53.503705 19500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:35505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:37465
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:48:53.505055 19500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:48:53.506968 19500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:48:53.521220 19505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:53.521387 19508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:53.521420 19506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:48:53.522354 19500 server_base.cc:1034] running on GCE node
I20260928 08:48:53.523089 19500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:48:53.524279 19500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:48:53.525543 19500 hybrid_clock.cc:648] HybridClock initialized: now 1790585333525495 us; error 43 us; skew 500 ppm
I20260928 08:48:53.525925 19500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:48:53.528503 19500 webserver.cc:460] Webserver started at http://127.0.97.3:43207/ using document root <none> and password file <none>
I20260928 08:48:53.529266 19500 fs_manager.cc:359] Metadata directory not provided
I20260928 08:48:53.529407 19500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:48:53.529760 19500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:48:53.532931 19500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-2/data/instance:
uuid: "e22e2f836ffe4f27a35b781ab52b2b7b"
format_stamp: "Formatted at 2026-09-28 08:48:53 on dist-test-slave-n6xs"
server_key: "76442c841e48ffb59d200fafdfe5c724"
server_key_iv: "5850d47053b1787d0419b56539d4c182"
server_key_version: "encryptionkey@0"
I20260928 08:48:53.533834 19500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-2/wal/instance:
uuid: "e22e2f836ffe4f27a35b781ab52b2b7b"
format_stamp: "Formatted at 2026-09-28 08:48:53 on dist-test-slave-n6xs"
server_key: "76442c841e48ffb59d200fafdfe5c724"
server_key_iv: "5850d47053b1787d0419b56539d4c182"
server_key_version: "encryptionkey@0"
I20260928 08:48:53.539886 19500 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260928 08:48:53.543740 19514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:53.545392 19500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:48:53.545598 19500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-2/wal
uuid: "e22e2f836ffe4f27a35b781ab52b2b7b"
format_stamp: "Formatted at 2026-09-28 08:48:53 on dist-test-slave-n6xs"
server_key: "76442c841e48ffb59d200fafdfe5c724"
server_key_iv: "5850d47053b1787d0419b56539d4c182"
server_key_version: "encryptionkey@0"
I20260928 08:48:53.545883 19500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:48:53.565692 19500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:48:53.566711 19500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:48:53.567028 19500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:48:53.568164 19500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:48:53.569746 19500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:48:53.569880 19500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:53.570017 19500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:48:53.570111 19500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:53.606089 19500 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:45807
I20260928 08:48:53.606204 19626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:45807 every 8 connection(s)
I20260928 08:48:53.607558 19500 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:48:53.614075 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 19500
I20260928 08:48:53.614296 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:48:53.614748 388 external_mini_cluster.cc:1428] Setting key 5c6e06ae3462d59fb70a2585f5cfed0e
I20260928 08:48:53.621398 19368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:53.610421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:35839 (local address 127.0.97.62:37465)
0928 08:48:53.610606 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:53.610611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:53.612136 (+ 1525us) server_negotiation.cc:407] Connection header received
0928 08:48:53.614055 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:53.614061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:53.614160 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:53.614278 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:53.615612 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:53.616542 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:53.618602 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:53.618977 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:53.619081 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:53.620662 (+ 1581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:53.620667 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:53.620670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:48:53.620690 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:53.620722 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:48:53.620734 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:53.620807 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:53.621036 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 08:48:53.621133 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 08:48:53.622601 19629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:53.610926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:37465 (local address 127.0.97.3:35839)
0928 08:48:53.611923 (+ 997us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:53.611961 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:53.613740 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:53.614534 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:53.614544 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:53.614644 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:48:53.615394 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:53.615423 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:53.616762 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:53.616769 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:53.618444 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:53.618460 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:53.619232 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:53.619238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:53.619480 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:53.620303 (+ 823us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:48:53.620306 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:48:53.620354 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:48:53.620453 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:48:53.620827 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:48:53.620835 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:48:53.620842 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 08:48:53.621009 (+ 167us) client_negotiation.cc:240] Negotiation successful
0928 08:48:53.621363 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":294,"threads_started":1}
I20260928 08:48:53.624293 19627 heartbeater.cc:347] Connected to a master server at 127.0.97.62:37465
I20260928 08:48:53.624657 19627 heartbeater.cc:464] Registering TS with master...
I20260928 08:48:53.625507 19627 heartbeater.cc:511] Master 127.0.97.62:37465 requested a full tablet report, sending...
I20260928 08:48:53.627194 19180 ts_manager.cc:194] Registered new tserver with Master: e22e2f836ffe4f27a35b781ab52b2b7b (127.0.97.3:45807)
I20260928 08:48:53.628129 19180 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:35839
I20260928 08:48:53.630445 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:48:53.642699 19368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:53.634845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37748 (local address 127.0.97.62:37465)
0928 08:48:53.635031 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:53.635036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:53.635538 (+ 502us) server_negotiation.cc:407] Connection header received
0928 08:48:53.635708 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:53.635713 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:53.635781 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:53.635967 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:53.636584 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:53.637778 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:53.639247 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:53.640005 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:53.640172 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:53.641462 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:53.641468 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:53.641473 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:48:53.641501 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:53.641543 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:48:53.641560 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:53.641740 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:53.642363 (+ 623us) server_negotiation.cc:299] Negotiation successful
0928 08:48:53.642430 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 08:48:53.647751 19180 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37748:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:48:53.655115 19180 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:48:53.667050 19366 heartbeater.cc:503] Master 127.0.97.62:37465 was elected leader, sending a full tablet report...
I20260928 08:48:53.692288 19641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:53.670126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:45807 (local address 127.0.0.1:36074)
0928 08:48:53.680429 (+ 10303us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:53.680467 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:53.680730 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:53.684533 (+ 3803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:53.684540 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:53.684581 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:48:53.685093 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:53.685113 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:53.687202 (+ 2089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:53.687209 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:53.689422 (+ 2213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:53.689440 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:53.691215 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:53.691221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:53.691446 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:53.691512 (+ 66us) client_negotiation.cc:769] Sending connection context
0928 08:48:53.691661 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 08:48:53.691896 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10050,"mutex_wait_us":68,"thread_start_us":10750,"threads_started":1}
I20260928 08:48:53.692272 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:53.670829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36074 (local address 127.0.97.3:45807)
0928 08:48:53.683382 (+ 12553us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:53.683391 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:53.683433 (+ 42us) server_negotiation.cc:407] Connection header received
0928 08:48:53.683548 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:53.683556 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:53.683852 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:53.684210 (+ 358us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:48:53.685358 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:53.687053 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:53.689733 (+ 2680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:53.690993 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:53.691175 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:53.691514 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:53.691706 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 08:48:53.691903 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12362,"thread_start_us":12213,"threads_started":1}
I20260928 08:48:53.693734 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:53.669506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51034 (local address 127.0.97.1:40741)
0928 08:48:53.682910 (+ 13404us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:53.682924 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:53.682971 (+ 47us) server_negotiation.cc:407] Connection header received
0928 08:48:53.683120 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:53.683130 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:53.683448 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:53.683725 (+ 277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:48:53.685130 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:53.686644 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:53.690499 (+ 3855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:53.692170 (+ 1671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:53.692336 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:53.692628 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:53.693199 (+ 571us) server_negotiation.cc:299] Negotiation successful
0928 08:48:53.693369 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13205,"thread_start_us":13052,"threads_started":1}
I20260928 08:48:53.694236 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:53.669395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40741 (local address 127.0.0.1:51034)
0928 08:48:53.670181 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:53.670227 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:53.670405 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:53.683837 (+ 13432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:53.683849 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:53.683902 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:48:53.684368 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:53.684384 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:53.686789 (+ 2405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:53.686800 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:53.690252 (+ 3452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:53.690267 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:53.692778 (+ 2511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:53.692786 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:53.692981 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:53.693022 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 08:48:53.693136 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 08:48:53.693297 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"mutex_wait_us":94,"thread_start_us":353,"threads_started":1}
I20260928 08:48:53.694880 19642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:53.670587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34999 (local address 127.0.0.1:47506)
0928 08:48:53.685055 (+ 14468us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:53.685084 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:53.685229 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:53.686329 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:53.686335 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:53.686362 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:48:53.686758 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:53.686771 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:53.688779 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:53.688785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:53.691176 (+ 2391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:53.691198 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:53.693621 (+ 2423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:53.693630 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:53.693817 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:53.694102 (+ 285us) client_negotiation.cc:769] Sending connection context
0928 08:48:53.694213 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 08:48:53.694384 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14280,"mutex_wait_us":226,"thread_start_us":14066,"threads_started":1}
I20260928 08:48:53.695577 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:53.671178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47506 (local address 127.0.97.2:34999)
0928 08:48:53.684488 (+ 13310us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:53.684497 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:53.685368 (+ 871us) server_negotiation.cc:407] Connection header received
0928 08:48:53.685499 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:53.685509 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:53.685817 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:53.686042 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:48:53.686941 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:53.688585 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:53.691468 (+ 2883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:53.693117 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:53.693258 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:53.693484 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:53.695146 (+ 1662us) server_negotiation.cc:299] Negotiation successful
0928 08:48:53.695300 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13147,"thread_start_us":13007,"threads_started":1}
I20260928 08:48:53.696969 19562 tablet_service.cc:1467] Processing CreateTablet for tablet 8ca1f34f97dd490f8f0f0918aa432e5f (DEFAULT_TABLE table=test-table [id=17a243e569764ba395040391aeb2d511]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:48:53.698256 19562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ca1f34f97dd490f8f0f0918aa432e5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:48:53.698259 19301 tablet_service.cc:1467] Processing CreateTablet for tablet 8ca1f34f97dd490f8f0f0918aa432e5f (DEFAULT_TABLE table=test-table [id=17a243e569764ba395040391aeb2d511]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:48:53.699868 19301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ca1f34f97dd490f8f0f0918aa432e5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:48:53.701087 19432 tablet_service.cc:1467] Processing CreateTablet for tablet 8ca1f34f97dd490f8f0f0918aa432e5f (DEFAULT_TABLE table=test-table [id=17a243e569764ba395040391aeb2d511]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:48:53.702454 19432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ca1f34f97dd490f8f0f0918aa432e5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:48:53.712176 19645 tablet_bootstrap.cc:492] T 8ca1f34f97dd490f8f0f0918aa432e5f P 389a8c16d22641a4ba3f4247d5200499: Bootstrap starting.
I20260928 08:48:53.714432 19646 tablet_bootstrap.cc:492] T 8ca1f34f97dd490f8f0f0918aa432e5f P e22e2f836ffe4f27a35b781ab52b2b7b: Bootstrap starting.
I20260928 08:48:53.717414 19647 tablet_bootstrap.cc:492] T 8ca1f34f97dd490f8f0f0918aa432e5f P b94a04c527ff40ff86ae8a15ad1fe82a: Bootstrap starting.
I20260928 08:48:53.718333 19646 tablet_bootstrap.cc:654] T 8ca1f34f97dd490f8f0f0918aa432e5f P e22e2f836ffe4f27a35b781ab52b2b7b: Neither blocks nor log segments found. Creating new log.
I20260928 08:48:53.718492 19645 tablet_bootstrap.cc:654] T 8ca1f34f97dd490f8f0f0918aa432e5f P 389a8c16d22641a4ba3f4247d5200499: Neither blocks nor log segments found. Creating new log.
I20260928 08:48:53.719910 19645 log.cc:824] T 8ca1f34f97dd490f8f0f0918aa432e5f P 389a8c16d22641a4ba3f4247d5200499: Log is configured to *not* fsync() on all Append() calls
I20260928 08:48:53.719910 19646 log.cc:824] T 8ca1f34f97dd490f8f0f0918aa432e5f P e22e2f836ffe4f27a35b781ab52b2b7b: Log is configured to *not* fsync() on all Append() calls
I20260928 08:48:53.721385 19647 tablet_bootstrap.cc:654] T 8ca1f34f97dd490f8f0f0918aa432e5f P b94a04c527ff40ff86ae8a15ad1fe82a: Neither blocks nor log segments found. Creating new log.
I20260928 08:48:53.722891 19647 log.cc:824] T 8ca1f34f97dd490f8f0f0918aa432e5f P b94a04c527ff40ff86ae8a15ad1fe82a: Log is configured to *not* fsync() on all Append() calls
I20260928 08:48:53.723536 19646 tablet_bootstrap.cc:492] T 8ca1f34f97dd490f8f0f0918aa432e5f P e22e2f836ffe4f27a35b781ab52b2b7b: No bootstrap required, opened a new log
I20260928 08:48:53.723955 19646 ts_tablet_manager.cc:1397] T 8ca1f34f97dd490f8f0f0918aa432e5f P e22e2f836ffe4f27a35b781ab52b2b7b: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20260928 08:48:53.724406 19645 tablet_bootstrap.cc:492] T 8ca1f34f97dd490f8f0f0918aa432e5f P 389a8c16d22641a4ba3f4247d5200499: No bootstrap required, opened a new log
I20260928 08:48:53.724694 19645 ts_tablet_manager.cc:1397] T 8ca1f34f97dd490f8f0f0918aa432e5f P 389a8c16d22641a4ba3f4247d5200499: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260928 08:48:53.726976 19647 tablet_bootstrap.cc:492] T 8ca1f34f97dd490f8f0f0918aa432e5f P b94a04c527ff40ff86ae8a15ad1fe82a: No bootstrap required, opened a new log
I20260928 08:48:53.727331 19647 ts_tablet_manager.cc:1397] T 8ca1f34f97dd490f8f0f0918aa432e5f P b94a04c527ff40ff86ae8a15ad1fe82a: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 08:48:53.732582 19646 raft_consensus.cc:348] T 8ca1f34f97dd490f8f0f0918aa432e5f P e22e2f836ffe4f27a35b781ab52b2b7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389a8c16d22641a4ba3f4247d5200499" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "e22e2f836ffe4f27a35b781ab52b2b7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45807 } } peers { permanent_uuid: "b94a04c527ff40ff86ae8a15ad1fe82a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34999 } }
I20260928 08:48:53.733166 19646 raft_consensus.cc:374] T 8ca1f34f97dd490f8f0f0918aa432e5f P e22e2f836ffe4f27a35b781ab52b2b7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:48:53.733357 19646 raft_consensus.cc:729] T 8ca1f34f97dd490f8f0f0918aa432e5f P e22e2f836ffe4f27a35b781ab52b2b7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e22e2f836ffe4f27a35b781ab52b2b7b, State: Initialized, Role: FOLLOWER
I20260928 08:48:53.734238 19646 consensus_queue.cc:260] T 8ca1f34f97dd490f8f0f0918aa432e5f P e22e2f836ffe4f27a35b781ab52b2b7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389a8c16d22641a4ba3f4247d5200499" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "e22e2f836ffe4f27a35b781ab52b2b7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45807 } } peers { permanent_uuid: "b94a04c527ff40ff86ae8a15ad1fe82a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34999 } }
I20260928 08:48:53.735847 19645 raft_consensus.cc:348] T 8ca1f34f97dd490f8f0f0918aa432e5f P 389a8c16d22641a4ba3f4247d5200499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389a8c16d22641a4ba3f4247d5200499" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "e22e2f836ffe4f27a35b781ab52b2b7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45807 } } peers { permanent_uuid: "b94a04c527ff40ff86ae8a15ad1fe82a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34999 } }
I20260928 08:48:53.736416 19645 raft_consensus.cc:374] T 8ca1f34f97dd490f8f0f0918aa432e5f P 389a8c16d22641a4ba3f4247d5200499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:48:53.736555 19645 raft_consensus.cc:729] T 8ca1f34f97dd490f8f0f0918aa432e5f P 389a8c16d22641a4ba3f4247d5200499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 389a8c16d22641a4ba3f4247d5200499, State: Initialized, Role: FOLLOWER
I20260928 08:48:53.736685 19627 heartbeater.cc:503] Master 127.0.97.62:37465 was elected leader, sending a full tablet report...
I20260928 08:48:53.737262 19645 consensus_queue.cc:260] T 8ca1f34f97dd490f8f0f0918aa432e5f P 389a8c16d22641a4ba3f4247d5200499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389a8c16d22641a4ba3f4247d5200499" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "e22e2f836ffe4f27a35b781ab52b2b7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45807 } } peers { permanent_uuid: "b94a04c527ff40ff86ae8a15ad1fe82a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34999 } }
I20260928 08:48:53.738400 19647 raft_consensus.cc:348] T 8ca1f34f97dd490f8f0f0918aa432e5f P b94a04c527ff40ff86ae8a15ad1fe82a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389a8c16d22641a4ba3f4247d5200499" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "e22e2f836ffe4f27a35b781ab52b2b7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45807 } } peers { permanent_uuid: "b94a04c527ff40ff86ae8a15ad1fe82a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34999 } }
I20260928 08:48:53.738965 19647 raft_consensus.cc:374] T 8ca1f34f97dd490f8f0f0918aa432e5f P b94a04c527ff40ff86ae8a15ad1fe82a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:48:53.739131 19647 raft_consensus.cc:729] T 8ca1f34f97dd490f8f0f0918aa432e5f P b94a04c527ff40ff86ae8a15ad1fe82a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94a04c527ff40ff86ae8a15ad1fe82a, State: Initialized, Role: FOLLOWER
I20260928 08:48:53.739753 19645 ts_tablet_manager.cc:1428] T 8ca1f34f97dd490f8f0f0918aa432e5f P 389a8c16d22641a4ba3f4247d5200499: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
I20260928 08:48:53.739980 19647 consensus_queue.cc:260] T 8ca1f34f97dd490f8f0f0918aa432e5f P b94a04c527ff40ff86ae8a15ad1fe82a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389a8c16d22641a4ba3f4247d5200499" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "e22e2f836ffe4f27a35b781ab52b2b7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45807 } } peers { permanent_uuid: "b94a04c527ff40ff86ae8a15ad1fe82a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34999 } }
I20260928 08:48:53.741319 19497 heartbeater.cc:503] Master 127.0.97.62:37465 was elected leader, sending a full tablet report...
I20260928 08:48:53.741953 19647 ts_tablet_manager.cc:1428] T 8ca1f34f97dd490f8f0f0918aa432e5f P b94a04c527ff40ff86ae8a15ad1fe82a: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260928 08:48:53.742197 19646 ts_tablet_manager.cc:1428] T 8ca1f34f97dd490f8f0f0918aa432e5f P e22e2f836ffe4f27a35b781ab52b2b7b: Time spent starting tablet: real 0.018s user 0.009s sys 0.003s
W20260928 08:48:53.859515 19628 tablet.cc:2369] T 8ca1f34f97dd490f8f0f0918aa432e5f P e22e2f836ffe4f27a35b781ab52b2b7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:48:53.868868 19651 raft_consensus.cc:482] T 8ca1f34f97dd490f8f0f0918aa432e5f P e22e2f836ffe4f27a35b781ab52b2b7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:48:53.869200 19651 raft_consensus.cc:504] T 8ca1f34f97dd490f8f0f0918aa432e5f P e22e2f836ffe4f27a35b781ab52b2b7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389a8c16d22641a4ba3f4247d5200499" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "e22e2f836ffe4f27a35b781ab52b2b7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45807 } } peers { permanent_uuid: "b94a04c527ff40ff86ae8a15ad1fe82a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34999 } }
I20260928 08:48:53.870913 19651 leader_election.cc:288] T 8ca1f34f97dd490f8f0f0918aa432e5f P e22e2f836ffe4f27a35b781ab52b2b7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 389a8c16d22641a4ba3f4247d5200499 (127.0.97.1:40741), b94a04c527ff40ff86ae8a15ad1fe82a (127.0.97.2:34999)
W20260928 08:48:53.876729 19498 tablet.cc:2369] T 8ca1f34f97dd490f8f0f0918aa432e5f P b94a04c527ff40ff86ae8a15ad1fe82a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:48:53.878394 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:53.871400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46281 (local address 127.0.97.1:40741)
0928 08:48:53.871633 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:53.871641 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:53.871740 (+ 99us) server_negotiation.cc:407] Connection header received
0928 08:48:53.872129 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:53.872134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:53.872226 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:53.872343 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:48:53.872828 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:53.873854 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:53.876095 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:53.877176 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:53.877306 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:53.877419 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:53.877990 (+ 571us) server_negotiation.cc:299] Negotiation successful
0928 08:48:53.878139 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 08:48:53.878448 19629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:53.871400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40741 (local address 127.0.97.3:46281)
0928 08:48:53.871617 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:53.871638 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:53.871799 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:53.872389 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:53.872394 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:53.872424 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:48:53.872613 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:53.872621 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:53.873969 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:53.873972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:53.875910 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:53.875925 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:53.877512 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:53.877562 (+ 50us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:53.877794 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:53.877836 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 08:48:53.877938 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 08:48:53.878092 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 08:48:53.879143 19654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:53.871848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:34999 (local address 127.0.97.3:42545)
0928 08:48:53.872540 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:53.872556 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:53.872701 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:53.873256 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:53.873260 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:53.873281 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:48:53.873599 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:53.873610 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:53.874667 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:53.874672 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:53.876586 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:53.876600 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:53.878331 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:53.878337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:53.878526 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:53.878564 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 08:48:53.878700 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 08:48:53.878849 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":393,"threads_started":1}
I20260928 08:48:53.879156 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:53.871971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:42545 (local address 127.0.97.2:34999)
0928 08:48:53.872166 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:53.872171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:53.872669 (+ 498us) server_negotiation.cc:407] Connection header received
0928 08:48:53.872981 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:53.872986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:53.873136 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:53.873231 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:48:53.873756 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:53.874518 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:53.876821 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:53.878054 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:53.878147 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:53.878231 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:53.878770 (+ 539us) server_negotiation.cc:299] Negotiation successful
0928 08:48:53.878905 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 08:48:53.879597 19321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca1f34f97dd490f8f0f0918aa432e5f" candidate_uuid: "e22e2f836ffe4f27a35b781ab52b2b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "389a8c16d22641a4ba3f4247d5200499" is_pre_election: true
I20260928 08:48:53.880337 19321 raft_consensus.cc:2454] T 8ca1f34f97dd490f8f0f0918aa432e5f P 389a8c16d22641a4ba3f4247d5200499 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e22e2f836ffe4f27a35b781ab52b2b7b in term 0.
I20260928 08:48:53.880270 19452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca1f34f97dd490f8f0f0918aa432e5f" candidate_uuid: "e22e2f836ffe4f27a35b781ab52b2b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94a04c527ff40ff86ae8a15ad1fe82a" is_pre_election: true
I20260928 08:48:53.880789 19452 raft_consensus.cc:2454] T 8ca1f34f97dd490f8f0f0918aa432e5f P b94a04c527ff40ff86ae8a15ad1fe82a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e22e2f836ffe4f27a35b781ab52b2b7b in term 0.
I20260928 08:48:53.881085 19515 leader_election.cc:302] T 8ca1f34f97dd490f8f0f0918aa432e5f P e22e2f836ffe4f27a35b781ab52b2b7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 389a8c16d22641a4ba3f4247d5200499, e22e2f836ffe4f27a35b781ab52b2b7b; no voters:
I20260928 08:48:53.881491 19651 raft_consensus.cc:2781] T 8ca1f34f97dd490f8f0f0918aa432e5f P e22e2f836ffe4f27a35b781ab52b2b7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:48:53.881620 19651 raft_consensus.cc:482] T 8ca1f34f97dd490f8f0f0918aa432e5f P e22e2f836ffe4f27a35b781ab52b2b7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:48:53.881743 19651 raft_consensus.cc:3037] T 8ca1f34f97dd490f8f0f0918aa432e5f P e22e2f836ffe4f27a35b781ab52b2b7b [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:48:53.884668 19651 raft_consensus.cc:504] T 8ca1f34f97dd490f8f0f0918aa432e5f P e22e2f836ffe4f27a35b781ab52b2b7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389a8c16d22641a4ba3f4247d5200499" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "e22e2f836ffe4f27a35b781ab52b2b7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45807 } } peers { permanent_uuid: "b94a04c527ff40ff86ae8a15ad1fe82a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34999 } }
I20260928 08:48:53.885293 19651 leader_election.cc:288] T 8ca1f34f97dd490f8f0f0918aa432e5f P e22e2f836ffe4f27a35b781ab52b2b7b [CANDIDATE]: Term 1 election: Requested vote from peers 389a8c16d22641a4ba3f4247d5200499 (127.0.97.1:40741), b94a04c527ff40ff86ae8a15ad1fe82a (127.0.97.2:34999)
I20260928 08:48:53.885952 19321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca1f34f97dd490f8f0f0918aa432e5f" candidate_uuid: "e22e2f836ffe4f27a35b781ab52b2b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "389a8c16d22641a4ba3f4247d5200499"
I20260928 08:48:53.885998 19452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ca1f34f97dd490f8f0f0918aa432e5f" candidate_uuid: "e22e2f836ffe4f27a35b781ab52b2b7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b94a04c527ff40ff86ae8a15ad1fe82a"
I20260928 08:48:53.886220 19321 raft_consensus.cc:3037] T 8ca1f34f97dd490f8f0f0918aa432e5f P 389a8c16d22641a4ba3f4247d5200499 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:48:53.886219 19452 raft_consensus.cc:3037] T 8ca1f34f97dd490f8f0f0918aa432e5f P b94a04c527ff40ff86ae8a15ad1fe82a [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:48:53.888665 19321 raft_consensus.cc:2454] T 8ca1f34f97dd490f8f0f0918aa432e5f P 389a8c16d22641a4ba3f4247d5200499 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e22e2f836ffe4f27a35b781ab52b2b7b in term 1.
I20260928 08:48:53.888675 19452 raft_consensus.cc:2454] T 8ca1f34f97dd490f8f0f0918aa432e5f P b94a04c527ff40ff86ae8a15ad1fe82a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e22e2f836ffe4f27a35b781ab52b2b7b in term 1.
I20260928 08:48:53.889246 19515 leader_election.cc:302] T 8ca1f34f97dd490f8f0f0918aa432e5f P e22e2f836ffe4f27a35b781ab52b2b7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b94a04c527ff40ff86ae8a15ad1fe82a, e22e2f836ffe4f27a35b781ab52b2b7b; no voters:
I20260928 08:48:53.889647 19651 raft_consensus.cc:2781] T 8ca1f34f97dd490f8f0f0918aa432e5f P e22e2f836ffe4f27a35b781ab52b2b7b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:48:53.890193 19651 raft_consensus.cc:686] T 8ca1f34f97dd490f8f0f0918aa432e5f P e22e2f836ffe4f27a35b781ab52b2b7b [term 1 LEADER]: Becoming Leader. State: Replica: e22e2f836ffe4f27a35b781ab52b2b7b, State: Running, Role: LEADER
W20260928 08:48:53.890934 19367 tablet.cc:2369] T 8ca1f34f97dd490f8f0f0918aa432e5f P 389a8c16d22641a4ba3f4247d5200499: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:48:53.890792 19651 consensus_queue.cc:237] T 8ca1f34f97dd490f8f0f0918aa432e5f P e22e2f836ffe4f27a35b781ab52b2b7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389a8c16d22641a4ba3f4247d5200499" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } } peers { permanent_uuid: "e22e2f836ffe4f27a35b781ab52b2b7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45807 } } peers { permanent_uuid: "b94a04c527ff40ff86ae8a15ad1fe82a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34999 } }
I20260928 08:48:53.895851 19179 catalog_manager.cc:5495] T 8ca1f34f97dd490f8f0f0918aa432e5f P e22e2f836ffe4f27a35b781ab52b2b7b reported cstate change: term changed from 0 to 1, leader changed from <none> to e22e2f836ffe4f27a35b781ab52b2b7b (127.0.97.3). New cstate: current_term: 1 leader_uuid: "e22e2f836ffe4f27a35b781ab52b2b7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "389a8c16d22641a4ba3f4247d5200499" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e22e2f836ffe4f27a35b781ab52b2b7b" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 45807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b94a04c527ff40ff86ae8a15ad1fe82a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34999 } health_report { overall_health: UNKNOWN } } }
I20260928 08:48:53.938161 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:53.932252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36078 (local address 127.0.97.3:45807)
0928 08:48:53.932436 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:53.932441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:53.932461 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:48:53.932701 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:53.932706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:53.932794 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:53.932985 (+ 191us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:48:53.933348 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:53.934081 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:53.935571 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:53.936089 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:53.936212 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:53.936872 (+ 660us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:48:53.937476 (+ 604us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:48:53.937640 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:53.937857 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 08:48:53.937914 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 08:48:53.947639 19452 raft_consensus.cc:1261] T 8ca1f34f97dd490f8f0f0918aa432e5f P b94a04c527ff40ff86ae8a15ad1fe82a [term 1 FOLLOWER]: Refusing update from remote peer e22e2f836ffe4f27a35b781ab52b2b7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:48:53.947640 19321 raft_consensus.cc:1261] T 8ca1f34f97dd490f8f0f0918aa432e5f P 389a8c16d22641a4ba3f4247d5200499 [term 1 FOLLOWER]: Refusing update from remote peer e22e2f836ffe4f27a35b781ab52b2b7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:48:53.948853 19655 consensus_queue.cc:1035] T 8ca1f34f97dd490f8f0f0918aa432e5f P e22e2f836ffe4f27a35b781ab52b2b7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "389a8c16d22641a4ba3f4247d5200499" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 40741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:48:53.949335 19651 consensus_queue.cc:1035] T 8ca1f34f97dd490f8f0f0918aa432e5f P e22e2f836ffe4f27a35b781ab52b2b7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b94a04c527ff40ff86ae8a15ad1fe82a" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 34999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:48:53.963964 19660 mvcc.cc:205] Tried to move back new op lower bound from 7334237527837986816 to 7334237527622062080. Current Snapshot: MvccSnapshot[applied={T|T < 7334237527837986816}]
I20260928 08:48:53.967237 19659 mvcc.cc:205] Tried to move back new op lower bound from 7334237527837986816 to 7334237527622062080. Current Snapshot: MvccSnapshot[applied={T|T < 7334237527837986816}]
I20260928 08:48:53.977421 19180 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:48:53.977771 19180 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37748:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:48:53.981954 19180 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2bc99e5bd7004181bbcf8b3d6cc0a70e: Sending DeleteTablet for 3 replicas of tablet 8ca1f34f97dd490f8f0f0918aa432e5f
I20260928 08:48:53.983265 19562 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ca1f34f97dd490f8f0f0918aa432e5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:48:53 UTC) from {username='slave'} at 127.0.0.1:36074
I20260928 08:48:53.983306 19301 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ca1f34f97dd490f8f0f0918aa432e5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:48:53 UTC) from {username='slave'} at 127.0.0.1:51034
I20260928 08:48:53.984365 19432 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ca1f34f97dd490f8f0f0918aa432e5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:48:53 UTC) from {username='slave'} at 127.0.0.1:47506
I20260928 08:48:53.984567 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 19239
I20260928 08:48:53.999040 19670 tablet_replica.cc:330] T 8ca1f34f97dd490f8f0f0918aa432e5f P b94a04c527ff40ff86ae8a15ad1fe82a: stopping tablet replica
I20260928 08:48:54.001292 19670 raft_consensus.cc:2229] T 8ca1f34f97dd490f8f0f0918aa432e5f P b94a04c527ff40ff86ae8a15ad1fe82a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:48:54.003655 19668 tablet_replica.cc:330] T 8ca1f34f97dd490f8f0f0918aa432e5f P 389a8c16d22641a4ba3f4247d5200499: stopping tablet replica
I20260928 08:48:54.004436 19669 tablet_replica.cc:330] T 8ca1f34f97dd490f8f0f0918aa432e5f P e22e2f836ffe4f27a35b781ab52b2b7b: stopping tablet replica
I20260928 08:48:54.004966 19669 raft_consensus.cc:2229] T 8ca1f34f97dd490f8f0f0918aa432e5f P e22e2f836ffe4f27a35b781ab52b2b7b [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:48:54.005754 19669 raft_consensus.cc:2258] T 8ca1f34f97dd490f8f0f0918aa432e5f P e22e2f836ffe4f27a35b781ab52b2b7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:48:54.006022 19670 raft_consensus.cc:2258] T 8ca1f34f97dd490f8f0f0918aa432e5f P b94a04c527ff40ff86ae8a15ad1fe82a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:48:54.006548 19668 raft_consensus.cc:2229] T 8ca1f34f97dd490f8f0f0918aa432e5f P 389a8c16d22641a4ba3f4247d5200499 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:48:54.007126 19668 raft_consensus.cc:2258] T 8ca1f34f97dd490f8f0f0918aa432e5f P 389a8c16d22641a4ba3f4247d5200499 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:48:54.009387 19668 ts_tablet_manager.cc:1905] T 8ca1f34f97dd490f8f0f0918aa432e5f P 389a8c16d22641a4ba3f4247d5200499: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:48:54.009438 19670 ts_tablet_manager.cc:1905] T 8ca1f34f97dd490f8f0f0918aa432e5f P b94a04c527ff40ff86ae8a15ad1fe82a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:48:54.010016 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:53.986134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51048 (local address 127.0.97.1:40741)
0928 08:48:53.986356 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:53.986364 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:53.997921 (+ 11557us) server_negotiation.cc:407] Connection header received
0928 08:48:53.999424 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:53.999435 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:53.999626 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:53.999747 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:54.001767 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:54.003389 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:54.005547 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:54.006253 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:54.006405 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:54.008197 (+ 1792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:54.008209 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:54.008227 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:48:54.009035 (+ 808us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:54.009106 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:48:54.009135 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:54.009256 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:54.009513 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 08:48:54.009646 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 08:48:54.011736 19361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:48:54.013837 19669 ts_tablet_manager.cc:1905] T 8ca1f34f97dd490f8f0f0918aa432e5f P e22e2f836ffe4f27a35b781ab52b2b7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:48:54.024653 19669 ts_tablet_manager.cc:1918] T 8ca1f34f97dd490f8f0f0918aa432e5f P e22e2f836ffe4f27a35b781ab52b2b7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:48:54.025128 19669 log.cc:1196] T 8ca1f34f97dd490f8f0f0918aa432e5f P e22e2f836ffe4f27a35b781ab52b2b7b: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-2/wal/wals/8ca1f34f97dd490f8f0f0918aa432e5f
I20260928 08:48:54.026010 19669 ts_tablet_manager.cc:1939] T 8ca1f34f97dd490f8f0f0918aa432e5f P e22e2f836ffe4f27a35b781ab52b2b7b: Deleting consensus metadata
I20260928 08:48:54.028764 19166 catalog_manager.cc:4841] TS e22e2f836ffe4f27a35b781ab52b2b7b (127.0.97.3:45807): tablet 8ca1f34f97dd490f8f0f0918aa432e5f (table test-table [id=17a243e569764ba395040391aeb2d511]) successfully deleted
I20260928 08:48:54.032670 19670 ts_tablet_manager.cc:1918] T 8ca1f34f97dd490f8f0f0918aa432e5f P b94a04c527ff40ff86ae8a15ad1fe82a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:48:54.033706 19670 log.cc:1196] T 8ca1f34f97dd490f8f0f0918aa432e5f P b94a04c527ff40ff86ae8a15ad1fe82a: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790585022442112-388-0/minicluster-data/ts-1/wal/wals/8ca1f34f97dd490f8f0f0918aa432e5f
I20260928 08:48:54.034652 19670 ts_tablet_manager.cc:1939] T 8ca1f34f97dd490f8f0f0918aa432e5f P b94a04c527ff40ff86ae8a15ad1fe82a: Deleting consensus metadata
I20260928 08:48:54.037438 19164 catalog_manager.cc:4841] TS b94a04c527ff40ff86ae8a15ad1fe82a (127.0.97.2:34999): tablet 8ca1f34f97dd490f8f0f0918aa432e5f (table test-table [id=17a243e569764ba395040391aeb2d511]) successfully deleted
I20260928 08:48:54.139919 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 19239
I20260928 08:48:54.161590 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 19370
W20260928 08:48:54.161756 19164 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:40741 (error 108)
I20260928 08:48:54.163784 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:54.163227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40741 (local address 127.0.0.1:51052)
0928 08:48:54.163487 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:54.163630 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260928 08:48:54.164120 19164 catalog_manager.cc:4584] TS 389a8c16d22641a4ba3f4247d5200499 (127.0.97.1:40741): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8ca1f34f97dd490f8f0f0918aa432e5f: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40741: connect: Connection refused (error 111)
I20260928 08:48:54.170047 19644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:54.162734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47508 (local address 127.0.97.2:34999)
0928 08:48:54.162936 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:54.162940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:54.162958 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:48:54.163012 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:54.163016 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:54.163095 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:54.163217 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:54.163685 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:54.164907 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:54.166350 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:54.167047 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:54.167179 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:54.168309 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:54.168321 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:54.168341 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:48:54.169192 (+ 851us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:54.169256 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:48:54.169283 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:54.169399 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:54.169535 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 08:48:54.169660 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 08:48:54.170807 19492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:48:54.217800 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:54.217326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40741 (local address 127.0.0.1:51060)
0928 08:48:54.217560 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:54.217645 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260928 08:48:54.218950 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:54.218585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40741 (local address 127.0.0.1:51064)
0928 08:48:54.218747 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:54.218833 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 08:48:54.279742 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:54.279233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40741 (local address 127.0.0.1:51074)
0928 08:48:54.279461 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:54.279561 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260928 08:48:54.280977 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:54.280603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40741 (local address 127.0.0.1:51088)
0928 08:48:54.280759 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:54.280860 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 08:48:54.320452 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 19370
I20260928 08:48:54.344765 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 19500
I20260928 08:48:54.361743 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:54.361240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40741 (local address 127.0.0.1:51090)
0928 08:48:54.361423 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:54.361522 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 08:48:54.361985 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:54.345676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36086 (local address 127.0.97.3:45807)
0928 08:48:54.345879 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:54.345884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:54.345904 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:48:54.346117 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:54.346121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:54.346208 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:54.346293 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:54.346796 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:54.347631 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:54.349598 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:54.350120 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:54.350239 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:54.360149 (+ 9910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:54.360164 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:54.360185 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:48:54.360995 (+ 810us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:54.361079 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:48:54.361112 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:54.361245 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:54.361486 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 08:48:54.361608 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 08:48:54.362890 19622 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:48:54.363420 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:54.362891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40741 (local address 127.0.0.1:51100)
0928 08:48:54.363209 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:54.363303 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199}
I20260928 08:48:54.490281 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 19500
I20260928 08:48:54.510617 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 19149
I20260928 08:48:54.511982 19210 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:48:54.623750 19639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:54.623068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:40741 (local address 127.0.0.1:51116)
0928 08:48:54.623325 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:54.623434 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:40741: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 08:48:54.623965 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 19149
2026-09-28T08:48:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3158 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:48:54 dist-test-slave-n6xs krb5kdc[19679](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:48:54 dist-test-slave-n6xs krb5kdc[19679](info): set up 2 sockets
Sep 28 08:48:54 dist-test-slave-n6xs krb5kdc[19679](info): commencing operation
krb5kdc: starting...
W20260928 08:48:55.792212 388 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:48:55 dist-test-slave-n6xs krb5kdc[19679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:48:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:48:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:48:56.064285 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42061
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42061
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:48:56.425338 19692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:48:56.425702 19692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:48:56.425897 19692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:48:56.433754 19692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:48:56.433895 19692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:48:56.433992 19692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:48:56.434085 19692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:48:56.434150 19692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:48:56.444762 19692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:48:56.446291 19692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:48:56.447753 19692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:48:56.461452 19697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:56.461403 19700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:56.461423 19698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:48:56.462488 19692 server_base.cc:1034] running on GCE node
I20260928 08:48:56.463428 19692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:48:56.465124 19692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:48:56.466372 19692 hybrid_clock.cc:648] HybridClock initialized: now 1790585336466310 us; error 58 us; skew 500 ppm
I20260928 08:48:56.466790 19692 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:48:56 dist-test-slave-n6xs krb5kdc[19679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585336, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:48:56.470355 19692 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:48:56.472177 19692 webserver.cc:460] Webserver started at http://127.0.97.62:44417/ using document root <none> and password file <none>
I20260928 08:48:56.473030 19692 fs_manager.cc:359] Metadata directory not provided
I20260928 08:48:56.473177 19692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:48:56.473543 19692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:48:56.477053 19692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "7d749479d2954260820ade10cd2ee5e4"
format_stamp: "Formatted at 2026-09-28 08:48:56 on dist-test-slave-n6xs"
server_key: "bd9cc5265c3a4d2acd1fc28a8d47e2df"
server_key_iv: "bd2dbfea1c27ef910c2d302e67c329ab"
server_key_version: "encryptionkey@0"
I20260928 08:48:56.477980 19692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "7d749479d2954260820ade10cd2ee5e4"
format_stamp: "Formatted at 2026-09-28 08:48:56 on dist-test-slave-n6xs"
server_key: "bd9cc5265c3a4d2acd1fc28a8d47e2df"
server_key_iv: "bd2dbfea1c27ef910c2d302e67c329ab"
server_key_version: "encryptionkey@0"
I20260928 08:48:56.484402 19692 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 08:48:56.488682 19707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:56.490345 19692 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 08:48:56.490621 19692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "7d749479d2954260820ade10cd2ee5e4"
format_stamp: "Formatted at 2026-09-28 08:48:56 on dist-test-slave-n6xs"
server_key: "bd9cc5265c3a4d2acd1fc28a8d47e2df"
server_key_iv: "bd2dbfea1c27ef910c2d302e67c329ab"
server_key_version: "encryptionkey@0"
I20260928 08:48:56.490851 19692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:48:56.514392 19692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:48:56.519302 19692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:48:56.519584 19692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:48:56.539769 19692 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42061
I20260928 08:48:56.539793 19758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42061 every 8 connection(s)
I20260928 08:48:56.541343 19692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:48:56.542447 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 19692
I20260928 08:48:56.542631 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:48:56.543112 388 external_mini_cluster.cc:1428] Setting key 97b6ef0c76106700e735e8a0a76dc8f5
I20260928 08:48:56.546196 19759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 08:48:56 dist-test-slave-n6xs krb5kdc[19679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:48:56.560690 19759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d749479d2954260820ade10cd2ee5e4: Bootstrap starting.
I20260928 08:48:56.564507 19759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d749479d2954260820ade10cd2ee5e4: Neither blocks nor log segments found. Creating new log.
I20260928 08:48:56.565696 19759 log.cc:824] T 00000000000000000000000000000000 P 7d749479d2954260820ade10cd2ee5e4: Log is configured to *not* fsync() on all Append() calls
I20260928 08:48:56.569196 19759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d749479d2954260820ade10cd2ee5e4: No bootstrap required, opened a new log
I20260928 08:48:56.569680 19762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:56.545181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39716 (local address 127.0.97.62:42061)
0928 08:48:56.546079 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:56.546096 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:56.546150 (+ 54us) server_negotiation.cc:407] Connection header received
0928 08:48:56.548622 (+ 2472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:56.548650 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:56.549165 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:56.549678 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:56.551313 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:56.553556 (+ 2243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:56.555032 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:56.556200 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:56.556481 (+ 281us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:56.561161 (+ 4680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:56.561170 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:56.561188 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:48:56.561242 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:56.564805 (+ 3563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:56.565655 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:56.565666 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:56.565685 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:56.565935 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:56.566366 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:56.566371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:56.566374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:56.567003 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:56.567343 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:56.567871 (+ 528us) server_negotiation.cc:299] Negotiation successful
0928 08:48:56.568357 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":395,"threads_started":1}
I20260928 08:48:56.577454 19759 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d749479d2954260820ade10cd2ee5e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d749479d2954260820ade10cd2ee5e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42061 } }
I20260928 08:48:56.577907 19759 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d749479d2954260820ade10cd2ee5e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:48:56.578059 19759 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d749479d2954260820ade10cd2ee5e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d749479d2954260820ade10cd2ee5e4, State: Initialized, Role: FOLLOWER
I20260928 08:48:56.578702 19759 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d749479d2954260820ade10cd2ee5e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d749479d2954260820ade10cd2ee5e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42061 } }
I20260928 08:48:56.578996 19759 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d749479d2954260820ade10cd2ee5e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:48:56.579169 19759 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d749479d2954260820ade10cd2ee5e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:48:56.579366 19759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d749479d2954260820ade10cd2ee5e4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:48:56.581751 19759 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d749479d2954260820ade10cd2ee5e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d749479d2954260820ade10cd2ee5e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42061 } }
I20260928 08:48:56.582335 19759 leader_election.cc:302] T 00000000000000000000000000000000 P 7d749479d2954260820ade10cd2ee5e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d749479d2954260820ade10cd2ee5e4; no voters:
I20260928 08:48:56.583042 19759 leader_election.cc:288] T 00000000000000000000000000000000 P 7d749479d2954260820ade10cd2ee5e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:48:56.583225 19764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d749479d2954260820ade10cd2ee5e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:48:56.584064 19764 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d749479d2954260820ade10cd2ee5e4 [term 1 LEADER]: Becoming Leader. State: Replica: 7d749479d2954260820ade10cd2ee5e4, State: Running, Role: LEADER
I20260928 08:48:56.584766 19764 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d749479d2954260820ade10cd2ee5e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d749479d2954260820ade10cd2ee5e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42061 } }
I20260928 08:48:56.585351 19759 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d749479d2954260820ade10cd2ee5e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:48:56.588176 19766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d749479d2954260820ade10cd2ee5e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d749479d2954260820ade10cd2ee5e4. Latest consensus state: current_term: 1 leader_uuid: "7d749479d2954260820ade10cd2ee5e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d749479d2954260820ade10cd2ee5e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42061 } } }
I20260928 08:48:56.588217 19765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d749479d2954260820ade10cd2ee5e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d749479d2954260820ade10cd2ee5e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d749479d2954260820ade10cd2ee5e4" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42061 } } }
I20260928 08:48:56.588752 19765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d749479d2954260820ade10cd2ee5e4 [sys.catalog]: This master's current role is: LEADER
I20260928 08:48:56.588757 19766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d749479d2954260820ade10cd2ee5e4 [sys.catalog]: This master's current role is: LEADER
I20260928 08:48:56.591346 19772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:48:56.596804 19772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:48:56.608240 19772 catalog_manager.cc:1347] Generated new cluster ID: 142f461ec10b4f689fa604f4df449273
I20260928 08:48:56.608422 19772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:48:56.622005 19772 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:48:56.623418 19772 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:48:56.631287 19772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d749479d2954260820ade10cd2ee5e4: Generated new TSK 0
I20260928 08:48:56.632543 19772 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.97.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
Entry for principal kudu/127.0.97.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab.
I20260928 08:48:56.699200 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42061
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:48:57.062350 19785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:48:57.062675 19785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:48:57.062850 19785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:48:57.070763 19785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:48:57.070899 19785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:48:57.071122 19785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:48:57.081748 19785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/krb5kdc/kudu_127.0.97.1.keytab
--principal=kudu/127.0.97.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:48:57.083191 19785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:48:57.084682 19785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:48:57.100127 19790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:57.100055 19791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:57.100108 19793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:48:57.101295 19785 server_base.cc:1034] running on GCE node
I20260928 08:48:57.102108 19785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:48:57.103185 19785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:48:57.104422 19785 hybrid_clock.cc:648] HybridClock initialized: now 1790585337104372 us; error 42 us; skew 500 ppm
I20260928 08:48:57.104851 19785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:48:57 dist-test-slave-n6xs krb5kdc[19679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:48:57.108855 19785 init.cc:377] Logged in from keytab as kudu/127.0.97.1@KRBTEST.COM (short username kudu)
I20260928 08:48:57.110773 19785 webserver.cc:460] Webserver started at http://127.0.97.1:37169/ using document root <none> and password file <none>
I20260928 08:48:57.111621 19785 fs_manager.cc:359] Metadata directory not provided
I20260928 08:48:57.111764 19785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:48:57.112180 19785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:48:57.115509 19785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-0/data/instance:
uuid: "888571675f4944c4b6b513528b43402a"
format_stamp: "Formatted at 2026-09-28 08:48:57 on dist-test-slave-n6xs"
server_key: "9b9e77c922c5ade1a1b417f9a737a9b5"
server_key_iv: "57098c6acca0d0a9ba13fecbbda259da"
server_key_version: "encryptionkey@0"
I20260928 08:48:57.116484 19785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-0/wal/instance:
uuid: "888571675f4944c4b6b513528b43402a"
format_stamp: "Formatted at 2026-09-28 08:48:57 on dist-test-slave-n6xs"
server_key: "9b9e77c922c5ade1a1b417f9a737a9b5"
server_key_iv: "57098c6acca0d0a9ba13fecbbda259da"
server_key_version: "encryptionkey@0"
I20260928 08:48:57.122673 19785 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 08:48:57.126982 19800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:57.128748 19785 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 08:48:57.128964 19785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-0/wal
uuid: "888571675f4944c4b6b513528b43402a"
format_stamp: "Formatted at 2026-09-28 08:48:57 on dist-test-slave-n6xs"
server_key: "9b9e77c922c5ade1a1b417f9a737a9b5"
server_key_iv: "57098c6acca0d0a9ba13fecbbda259da"
server_key_version: "encryptionkey@0"
I20260928 08:48:57.129215 19785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:48:57.144927 19785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:48:57.149364 19785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:48:57.149706 19785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:48:57.150775 19785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:48:57.152329 19785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:48:57.152410 19785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:57.152562 19785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:48:57.152619 19785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:57.187780 19785 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:39937
I20260928 08:48:57.187913 19912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:39937 every 8 connection(s)
I20260928 08:48:57.189247 19785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:48:57.196789 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 19785
I20260928 08:48:57.196986 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:48:57.197430 388 external_mini_cluster.cc:1428] Setting key b1b45de308ef87cb8b9e3dd38d1d839f
Sep 28 08:48:57 dist-test-slave-n6xs krb5kdc[19679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.1@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:48:57.214816 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:57.192121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34133 (local address 127.0.97.62:42061)
0928 08:48:57.192630 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:57.192635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:57.193541 (+ 906us) server_negotiation.cc:407] Connection header received
0928 08:48:57.195634 (+ 2093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:57.195640 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:57.195743 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:57.195953 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:57.197841 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:57.198868 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:57.201637 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:57.202106 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:57.202240 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:57.206996 (+ 4756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:57.207006 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:57.207014 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:48:57.207071 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:57.210566 (+ 3495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:57.211912 (+ 1346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:57.211917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:57.211919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:57.212083 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:57.212719 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:57.212724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:57.212726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:57.213082 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:57.213203 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:57.214347 (+ 1144us) server_negotiation.cc:299] Negotiation successful
0928 08:48:57.214549 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":267,"threads_started":1}
I20260928 08:48:57.216668 19916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:57.192603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42061 (local address 127.0.97.1:34133)
0928 08:48:57.193346 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:57.193400 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:57.195306 (+ 1906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:57.196231 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:57.196241 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:57.196886 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:48:57.197648 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:57.197669 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:57.199070 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:57.199074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:57.201410 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:57.201434 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:57.202394 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:57.202404 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:57.202786 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:57.203667 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:48:57.203705 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:48:57.206664 (+ 2959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:48:57.210805 (+ 4141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:48:57.210820 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:48:57.210845 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:48:57.211676 (+ 831us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:48:57.212212 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:48:57.212222 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:48:57.212230 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:48:57.212549 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:48:57.213254 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:48:57.213267 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:48:57.213833 (+ 566us) client_negotiation.cc:769] Sending connection context
0928 08:48:57.214213 (+ 380us) client_negotiation.cc:240] Negotiation successful
0928 08:48:57.214915 (+ 702us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":287,"threads_started":1}
I20260928 08:48:57.218989 19913 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42061
I20260928 08:48:57.219359 19913 heartbeater.cc:464] Registering TS with master...
I20260928 08:48:57.220256 19913 heartbeater.cc:511] Master 127.0.97.62:42061 requested a full tablet report, sending...
I20260928 08:48:57.222703 19724 ts_manager.cc:194] Registered new tserver with Master: 888571675f4944c4b6b513528b43402a (127.0.97.1:39937)
I20260928 08:48:57.224433 19724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.1@KRBTEST.COM'} at 127.0.97.1:34133
WARNING: no policy specified for kudu/127.0.97.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
Entry for principal kudu/127.0.97.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab.
I20260928 08:48:57.261158 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42061
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:48:57.616685 19919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:48:57.617054 19919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:48:57.617233 19919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:48:57.625160 19919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:48:57.625372 19919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:48:57.625578 19919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:48:57.637717 19919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/krb5kdc/kudu_127.0.97.2.keytab
--principal=kudu/127.0.97.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:48:57.639251 19919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:48:57.640908 19919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:48:57.657027 19925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:57.657127 19927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:57.657267 19924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:48:57.658581 19919 server_base.cc:1034] running on GCE node
I20260928 08:48:57.659392 19919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:48:57.660602 19919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:48:57.661862 19919 hybrid_clock.cc:648] HybridClock initialized: now 1790585337661808 us; error 53 us; skew 500 ppm
I20260928 08:48:57.662278 19919 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:48:57 dist-test-slave-n6xs krb5kdc[19679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:48:57.665899 19919 init.cc:377] Logged in from keytab as kudu/127.0.97.2@KRBTEST.COM (short username kudu)
I20260928 08:48:57.667771 19919 webserver.cc:460] Webserver started at http://127.0.97.2:38241/ using document root <none> and password file <none>
I20260928 08:48:57.668628 19919 fs_manager.cc:359] Metadata directory not provided
I20260928 08:48:57.668787 19919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:48:57.669131 19919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:48:57.672461 19919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-1/data/instance:
uuid: "78d7400d33184734835817f4615ab778"
format_stamp: "Formatted at 2026-09-28 08:48:57 on dist-test-slave-n6xs"
server_key: "d95ce0825b621c1ebf0173a2b069f982"
server_key_iv: "d652fdce37b6c3ee27eba4ccc474cc9a"
server_key_version: "encryptionkey@0"
I20260928 08:48:57.673367 19919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-1/wal/instance:
uuid: "78d7400d33184734835817f4615ab778"
format_stamp: "Formatted at 2026-09-28 08:48:57 on dist-test-slave-n6xs"
server_key: "d95ce0825b621c1ebf0173a2b069f982"
server_key_iv: "d652fdce37b6c3ee27eba4ccc474cc9a"
server_key_version: "encryptionkey@0"
I20260928 08:48:57.679577 19919 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 08:48:57.683791 19934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:57.685520 19919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:48:57.685741 19919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-1/wal
uuid: "78d7400d33184734835817f4615ab778"
format_stamp: "Formatted at 2026-09-28 08:48:57 on dist-test-slave-n6xs"
server_key: "d95ce0825b621c1ebf0173a2b069f982"
server_key_iv: "d652fdce37b6c3ee27eba4ccc474cc9a"
server_key_version: "encryptionkey@0"
I20260928 08:48:57.685987 19919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:48:57.723706 19919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:48:57.728808 19919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:48:57.729149 19919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:48:57.730312 19919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:48:57.732038 19919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:48:57.732149 19919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:57.732314 19919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:48:57.732390 19919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:57.769167 19919 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:38551
I20260928 08:48:57.769202 20046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:38551 every 8 connection(s)
I20260928 08:48:57.770707 19919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:48:57.779917 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 19919
I20260928 08:48:57.780131 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:48:57.780632 388 external_mini_cluster.cc:1428] Setting key f376caa871483634952b59889a43d3a8
Sep 28 08:48:57 dist-test-slave-n6xs krb5kdc[19679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.2@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:48:57.795060 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:57.773859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:45455 (local address 127.0.97.62:42061)
0928 08:48:57.774397 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:57.774404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:57.775482 (+ 1078us) server_negotiation.cc:407] Connection header received
0928 08:48:57.777566 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:57.777573 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:57.777681 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:57.777806 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:57.779579 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:57.780531 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:57.782422 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:57.783049 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:57.783205 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:57.787857 (+ 4652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:57.787871 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:57.787874 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:48:57.787923 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:57.791247 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:57.792313 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:57.792322 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:57.792332 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:57.792509 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:57.793028 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:57.793032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:57.793035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:57.793333 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:57.793526 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:57.794307 (+ 781us) server_negotiation.cc:299] Negotiation successful
0928 08:48:57.794650 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":264,"threads_started":1}
I20260928 08:48:57.796252 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:57.774503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42061 (local address 127.0.97.2:45455)
0928 08:48:57.775268 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:57.775316 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:57.777215 (+ 1899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:57.778088 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:57.778098 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:57.778730 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:48:57.779344 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:57.779363 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:57.780714 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:57.780718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:57.782259 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:57.782274 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:57.783317 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:57.783325 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:57.783589 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:57.784438 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:48:57.784469 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:48:57.787465 (+ 2996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:48:57.791464 (+ 3999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:48:57.791479 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:48:57.791502 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:48:57.792141 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:48:57.792661 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:48:57.792664 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:48:57.792667 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:48:57.792874 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:48:57.793497 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:48:57.793507 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:48:57.793903 (+ 396us) client_negotiation.cc:769] Sending connection context
0928 08:48:57.794201 (+ 298us) client_negotiation.cc:240] Negotiation successful
0928 08:48:57.794864 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":306,"threads_started":1}
I20260928 08:48:57.798094 20047 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42061
I20260928 08:48:57.798512 20047 heartbeater.cc:464] Registering TS with master...
I20260928 08:48:57.799567 20047 heartbeater.cc:511] Master 127.0.97.62:42061 requested a full tablet report, sending...
I20260928 08:48:57.801779 19724 ts_manager.cc:194] Registered new tserver with Master: 78d7400d33184734835817f4615ab778 (127.0.97.2:38551)
I20260928 08:48:57.803164 19724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.2@KRBTEST.COM'} at 127.0.97.2:45455
WARNING: no policy specified for kudu/127.0.97.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
Entry for principal kudu/127.0.97.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab.
I20260928 08:48:57.843107 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42061
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:33043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:48:58.200637 20053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:48:58.200953 20053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:48:58.201139 20053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:48:58.208930 20053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:48:58.209074 20053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:48:58.209265 20053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:48:58.219504 20053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/krb5kdc/kudu_127.0.97.3.keytab
--principal=kudu/127.0.97.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:42061
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:48:58.220943 20053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:48:58.222366 20053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 08:48:58.228055 19913 heartbeater.cc:503] Master 127.0.97.62:42061 was elected leader, sending a full tablet report...
W20260928 08:48:58.236938 20058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:58.236857 20061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:58.236876 20059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:48:58.237895 20053 server_base.cc:1034] running on GCE node
I20260928 08:48:58.238608 20053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:48:58.239647 20053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:48:58.240868 20053 hybrid_clock.cc:648] HybridClock initialized: now 1790585338240792 us; error 76 us; skew 500 ppm
I20260928 08:48:58.241194 20053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:48:58 dist-test-slave-n6xs krb5kdc[19679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:48:58.244717 20053 init.cc:377] Logged in from keytab as kudu/127.0.97.3@KRBTEST.COM (short username kudu)
I20260928 08:48:58.246356 20053 webserver.cc:460] Webserver started at http://127.0.97.3:44505/ using document root <none> and password file <none>
I20260928 08:48:58.247076 20053 fs_manager.cc:359] Metadata directory not provided
I20260928 08:48:58.247198 20053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:48:58.247458 20053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:48:58.250639 20053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-2/data/instance:
uuid: "0d725ccac088474ba2cf1848c236d8cc"
format_stamp: "Formatted at 2026-09-28 08:48:58 on dist-test-slave-n6xs"
server_key: "35069744cd65f6d71455550c3b4cfbd1"
server_key_iv: "2562a810f3ef5e2ca3a5a75659cd58e3"
server_key_version: "encryptionkey@0"
I20260928 08:48:58.251636 20053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-2/wal/instance:
uuid: "0d725ccac088474ba2cf1848c236d8cc"
format_stamp: "Formatted at 2026-09-28 08:48:58 on dist-test-slave-n6xs"
server_key: "35069744cd65f6d71455550c3b4cfbd1"
server_key_iv: "2562a810f3ef5e2ca3a5a75659cd58e3"
server_key_version: "encryptionkey@0"
I20260928 08:48:58.257395 20053 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 08:48:58.261292 20068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:58.262863 20053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:48:58.263070 20053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-2/wal
uuid: "0d725ccac088474ba2cf1848c236d8cc"
format_stamp: "Formatted at 2026-09-28 08:48:58 on dist-test-slave-n6xs"
server_key: "35069744cd65f6d71455550c3b4cfbd1"
server_key_iv: "2562a810f3ef5e2ca3a5a75659cd58e3"
server_key_version: "encryptionkey@0"
I20260928 08:48:58.263263 20053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:48:58.284037 20053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:48:58.289641 20053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:48:58.289909 20053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:48:58.290984 20053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:48:58.292776 20053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:48:58.292862 20053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:58.292958 20053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:48:58.293033 20053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:58.328524 20053 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:46103
I20260928 08:48:58.328570 20180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:46103 every 8 connection(s)
I20260928 08:48:58.329918 20053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:48:58.337478 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 20053
I20260928 08:48:58.337671 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:48:58.338187 388 external_mini_cluster.cc:1428] Setting key 1f2cbd6ee74fdcfd3e7f7f261166d1fb
Sep 28 08:48:58 dist-test-slave-n6xs krb5kdc[19679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585338, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.3@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:48:58.352222 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:58.332770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:47419 (local address 127.0.97.62:42061)
0928 08:48:58.333222 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:58.333227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:58.334454 (+ 1227us) server_negotiation.cc:407] Connection header received
0928 08:48:58.336486 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:58.336491 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:58.336583 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:58.336680 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:58.338924 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:58.340071 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:58.342091 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:58.342496 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:58.342604 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:58.346376 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:58.346382 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:58.346385 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:48:58.346429 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:58.349236 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:58.349969 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:58.349974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:58.349978 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:58.350098 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:58.350461 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:58.350464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:58.350466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:58.350730 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:58.350841 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:58.351581 (+ 740us) server_negotiation.cc:299] Negotiation successful
0928 08:48:58.351776 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":240,"threads_started":1}
I20260928 08:48:58.353482 20184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:58.333322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:42061 (local address 127.0.97.3:47419)
0928 08:48:58.334255 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:58.334326 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:58.336159 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:58.336908 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:58.336917 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:58.337805 (+ 888us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:48:58.338682 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:58.338715 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:58.340283 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:58.340291 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:58.341939 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:58.341953 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:58.342734 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:58.342741 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:58.343002 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:58.343716 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:48:58.343745 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:48:58.346144 (+ 2399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:48:58.349379 (+ 3235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:48:58.349388 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:48:58.349404 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:48:58.349852 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:48:58.350186 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:48:58.350189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:48:58.350191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:48:58.350370 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:48:58.350831 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:48:58.350837 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:48:58.351224 (+ 387us) client_negotiation.cc:769] Sending connection context
0928 08:48:58.351474 (+ 250us) client_negotiation.cc:240] Negotiation successful
0928 08:48:58.352045 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":356,"threads_started":1}
I20260928 08:48:58.354929 20181 heartbeater.cc:347] Connected to a master server at 127.0.97.62:42061
I20260928 08:48:58.355283 20181 heartbeater.cc:464] Registering TS with master...
I20260928 08:48:58.356153 20181 heartbeater.cc:511] Master 127.0.97.62:42061 requested a full tablet report, sending...
I20260928 08:48:58.357661 19724 ts_manager.cc:194] Registered new tserver with Master: 0d725ccac088474ba2cf1848c236d8cc (127.0.97.3:46103)
I20260928 08:48:58.358502 19724 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.97.3@KRBTEST.COM'} at 127.0.97.3:47419
I20260928 08:48:58.366773 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:48:58.384595 20183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:58.371413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39724 (local address 127.0.97.62:42061)
0928 08:48:58.371701 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:58.371708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:58.372086 (+ 378us) server_negotiation.cc:407] Connection header received
0928 08:48:58.372290 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:58.372295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:58.372379 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:58.372488 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:58.374003 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:58.374768 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:58.376210 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:58.376651 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:58.376767 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:58.379134 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:58.379139 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:58.379143 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:48:58.379182 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:58.382021 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:58.382740 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:58.382744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:58.382746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:58.382879 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:58.383202 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:58.383205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:58.383207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:58.383557 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:58.383745 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:58.384133 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 08:48:58.384321 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260928 08:48:58.388772 19724 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39724:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:48:58.396390 19724 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:48:58.433156 20194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:58.409520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:46103 (local address 127.0.0.1:43550)
0928 08:48:58.423740 (+ 14220us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:58.423790 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:58.424094 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:58.424897 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:58.424907 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:58.424948 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:48:58.425341 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:58.425358 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:58.427693 (+ 2335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:58.427701 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:58.429739 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:58.429755 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:58.431475 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:58.431481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:58.431717 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:58.431766 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 08:48:58.431958 (+ 192us) client_negotiation.cc:240] Negotiation successful
0928 08:48:58.432182 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13893,"thread_start_us":13730,"threads_started":1}
I20260928 08:48:58.433233 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:58.409897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43550 (local address 127.0.97.3:46103)
0928 08:48:58.423211 (+ 13314us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:58.423223 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:58.423972 (+ 749us) server_negotiation.cc:407] Connection header received
0928 08:48:58.424329 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:58.424341 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:58.424650 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:58.425544 (+ 894us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:48:58.425838 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:58.427550 (+ 1712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:58.429946 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:58.431275 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:58.431423 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:58.431679 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:58.432677 (+ 998us) server_negotiation.cc:299] Negotiation successful
0928 08:48:58.432861 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13102,"thread_start_us":12923,"threads_started":1}
I20260928 08:48:58.433951 20197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:58.410538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42480 (local address 127.0.97.2:38551)
0928 08:48:58.423638 (+ 13100us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:58.423645 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:58.424980 (+ 1335us) server_negotiation.cc:407] Connection header received
0928 08:48:58.425083 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:58.425090 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:58.425326 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:58.425530 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:48:58.426550 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:58.428097 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:58.430776 (+ 2679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:58.432202 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:58.432344 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:58.432560 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:58.433376 (+ 816us) server_negotiation.cc:299] Negotiation successful
0928 08:48:58.433546 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12923,"thread_start_us":13091,"threads_started":1}
I20260928 08:48:58.434032 20196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:58.410238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38551 (local address 127.0.0.1:42480)
0928 08:48:58.424710 (+ 14472us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:58.424739 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:58.424857 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:58.425811 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:58.425816 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:58.425840 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:48:58.426340 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:58.426354 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:58.428275 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:58.428280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:58.430584 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:58.430603 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:58.432420 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:58.432426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:58.432668 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:58.432701 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 08:48:58.432786 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 08:48:58.432912 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14345,"thread_start_us":14165,"threads_started":1}
I20260928 08:48:58.434813 20198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:58.425104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33286 (local address 127.0.97.1:39937)
0928 08:48:58.425771 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:58.425778 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:58.426531 (+ 753us) server_negotiation.cc:407] Connection header received
0928 08:48:58.426814 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:58.426820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:58.426977 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:58.427105 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:48:58.427797 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:58.429410 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:58.431945 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:58.433357 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:58.433482 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:58.433724 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:58.434324 (+ 600us) server_negotiation.cc:299] Negotiation successful
0928 08:48:58.434494 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":339,"threads_started":1}
I20260928 08:48:58.437445 19982 tablet_service.cc:1467] Processing CreateTablet for tablet e7a975c160be4b5b8526c2aec53cc6b7 (DEFAULT_TABLE table=test-table [id=f4f2aef01a2e461f9fb24bbdbe0a509a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:48:58.437446 20116 tablet_service.cc:1467] Processing CreateTablet for tablet e7a975c160be4b5b8526c2aec53cc6b7 (DEFAULT_TABLE table=test-table [id=f4f2aef01a2e461f9fb24bbdbe0a509a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:48:58.438297 20199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:58.425808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39937 (local address 127.0.0.1:33286)
0928 08:48:58.426428 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:58.426445 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:58.426631 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:58.427316 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:58.427319 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:58.427338 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:48:58.427598 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:58.427605 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:58.429583 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:58.429589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:58.431742 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:58.431756 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:58.433941 (+ 2185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:58.433947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:58.434119 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:58.434150 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 08:48:58.435064 (+ 914us) client_negotiation.cc:240] Negotiation successful
0928 08:48:58.437567 (+ 2503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":344,"threads_started":1}
I20260928 08:48:58.438926 20116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7a975c160be4b5b8526c2aec53cc6b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:48:58.438951 19982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7a975c160be4b5b8526c2aec53cc6b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:48:58.442605 19848 tablet_service.cc:1467] Processing CreateTablet for tablet e7a975c160be4b5b8526c2aec53cc6b7 (DEFAULT_TABLE table=test-table [id=f4f2aef01a2e461f9fb24bbdbe0a509a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:48:58.444756 19848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7a975c160be4b5b8526c2aec53cc6b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:48:58.451050 20200 tablet_bootstrap.cc:492] T e7a975c160be4b5b8526c2aec53cc6b7 P 78d7400d33184734835817f4615ab778: Bootstrap starting.
I20260928 08:48:58.453595 20201 tablet_bootstrap.cc:492] T e7a975c160be4b5b8526c2aec53cc6b7 P 0d725ccac088474ba2cf1848c236d8cc: Bootstrap starting.
I20260928 08:48:58.454929 20200 tablet_bootstrap.cc:654] T e7a975c160be4b5b8526c2aec53cc6b7 P 78d7400d33184734835817f4615ab778: Neither blocks nor log segments found. Creating new log.
I20260928 08:48:58.456413 20200 log.cc:824] T e7a975c160be4b5b8526c2aec53cc6b7 P 78d7400d33184734835817f4615ab778: Log is configured to *not* fsync() on all Append() calls
I20260928 08:48:58.457525 20201 tablet_bootstrap.cc:654] T e7a975c160be4b5b8526c2aec53cc6b7 P 0d725ccac088474ba2cf1848c236d8cc: Neither blocks nor log segments found. Creating new log.
I20260928 08:48:58.458823 20201 log.cc:824] T e7a975c160be4b5b8526c2aec53cc6b7 P 0d725ccac088474ba2cf1848c236d8cc: Log is configured to *not* fsync() on all Append() calls
I20260928 08:48:58.460019 20203 tablet_bootstrap.cc:492] T e7a975c160be4b5b8526c2aec53cc6b7 P 888571675f4944c4b6b513528b43402a: Bootstrap starting.
I20260928 08:48:58.460466 20200 tablet_bootstrap.cc:492] T e7a975c160be4b5b8526c2aec53cc6b7 P 78d7400d33184734835817f4615ab778: No bootstrap required, opened a new log
I20260928 08:48:58.460779 20200 ts_tablet_manager.cc:1397] T e7a975c160be4b5b8526c2aec53cc6b7 P 78d7400d33184734835817f4615ab778: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260928 08:48:58.462996 20201 tablet_bootstrap.cc:492] T e7a975c160be4b5b8526c2aec53cc6b7 P 0d725ccac088474ba2cf1848c236d8cc: No bootstrap required, opened a new log
I20260928 08:48:58.463287 20201 ts_tablet_manager.cc:1397] T e7a975c160be4b5b8526c2aec53cc6b7 P 0d725ccac088474ba2cf1848c236d8cc: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 08:48:58.464458 20203 tablet_bootstrap.cc:654] T e7a975c160be4b5b8526c2aec53cc6b7 P 888571675f4944c4b6b513528b43402a: Neither blocks nor log segments found. Creating new log.
I20260928 08:48:58.465920 20203 log.cc:824] T e7a975c160be4b5b8526c2aec53cc6b7 P 888571675f4944c4b6b513528b43402a: Log is configured to *not* fsync() on all Append() calls
I20260928 08:48:58.469487 20203 tablet_bootstrap.cc:492] T e7a975c160be4b5b8526c2aec53cc6b7 P 888571675f4944c4b6b513528b43402a: No bootstrap required, opened a new log
I20260928 08:48:58.469839 20203 ts_tablet_manager.cc:1397] T e7a975c160be4b5b8526c2aec53cc6b7 P 888571675f4944c4b6b513528b43402a: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260928 08:48:58.469604 20200 raft_consensus.cc:348] T e7a975c160be4b5b8526c2aec53cc6b7 P 78d7400d33184734835817f4615ab778 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d725ccac088474ba2cf1848c236d8cc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46103 } } peers { permanent_uuid: "888571675f4944c4b6b513528b43402a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39937 } } peers { permanent_uuid: "78d7400d33184734835817f4615ab778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38551 } }
I20260928 08:48:58.470086 20200 raft_consensus.cc:374] T e7a975c160be4b5b8526c2aec53cc6b7 P 78d7400d33184734835817f4615ab778 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:48:58.470218 20200 raft_consensus.cc:729] T e7a975c160be4b5b8526c2aec53cc6b7 P 78d7400d33184734835817f4615ab778 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78d7400d33184734835817f4615ab778, State: Initialized, Role: FOLLOWER
I20260928 08:48:58.470845 20200 consensus_queue.cc:260] T e7a975c160be4b5b8526c2aec53cc6b7 P 78d7400d33184734835817f4615ab778 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d725ccac088474ba2cf1848c236d8cc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46103 } } peers { permanent_uuid: "888571675f4944c4b6b513528b43402a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39937 } } peers { permanent_uuid: "78d7400d33184734835817f4615ab778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38551 } }
I20260928 08:48:58.471976 20047 heartbeater.cc:503] Master 127.0.97.62:42061 was elected leader, sending a full tablet report...
I20260928 08:48:58.472523 20200 ts_tablet_manager.cc:1428] T e7a975c160be4b5b8526c2aec53cc6b7 P 78d7400d33184734835817f4615ab778: Time spent starting tablet: real 0.012s user 0.009s sys 0.003s
I20260928 08:48:58.473948 20201 raft_consensus.cc:348] T e7a975c160be4b5b8526c2aec53cc6b7 P 0d725ccac088474ba2cf1848c236d8cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d725ccac088474ba2cf1848c236d8cc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46103 } } peers { permanent_uuid: "888571675f4944c4b6b513528b43402a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39937 } } peers { permanent_uuid: "78d7400d33184734835817f4615ab778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38551 } }
I20260928 08:48:58.474334 20201 raft_consensus.cc:374] T e7a975c160be4b5b8526c2aec53cc6b7 P 0d725ccac088474ba2cf1848c236d8cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:48:58.474449 20201 raft_consensus.cc:729] T e7a975c160be4b5b8526c2aec53cc6b7 P 0d725ccac088474ba2cf1848c236d8cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d725ccac088474ba2cf1848c236d8cc, State: Initialized, Role: FOLLOWER
I20260928 08:48:58.475066 20201 consensus_queue.cc:260] T e7a975c160be4b5b8526c2aec53cc6b7 P 0d725ccac088474ba2cf1848c236d8cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d725ccac088474ba2cf1848c236d8cc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46103 } } peers { permanent_uuid: "888571675f4944c4b6b513528b43402a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39937 } } peers { permanent_uuid: "78d7400d33184734835817f4615ab778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38551 } }
I20260928 08:48:58.476215 20181 heartbeater.cc:503] Master 127.0.97.62:42061 was elected leader, sending a full tablet report...
I20260928 08:48:58.478508 20201 ts_tablet_manager.cc:1428] T e7a975c160be4b5b8526c2aec53cc6b7 P 0d725ccac088474ba2cf1848c236d8cc: Time spent starting tablet: real 0.015s user 0.010s sys 0.005s
I20260928 08:48:58.480309 20203 raft_consensus.cc:348] T e7a975c160be4b5b8526c2aec53cc6b7 P 888571675f4944c4b6b513528b43402a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d725ccac088474ba2cf1848c236d8cc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46103 } } peers { permanent_uuid: "888571675f4944c4b6b513528b43402a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39937 } } peers { permanent_uuid: "78d7400d33184734835817f4615ab778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38551 } }
I20260928 08:48:58.480841 20203 raft_consensus.cc:374] T e7a975c160be4b5b8526c2aec53cc6b7 P 888571675f4944c4b6b513528b43402a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:48:58.480971 20203 raft_consensus.cc:729] T e7a975c160be4b5b8526c2aec53cc6b7 P 888571675f4944c4b6b513528b43402a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 888571675f4944c4b6b513528b43402a, State: Initialized, Role: FOLLOWER
I20260928 08:48:58.481560 20203 consensus_queue.cc:260] T e7a975c160be4b5b8526c2aec53cc6b7 P 888571675f4944c4b6b513528b43402a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d725ccac088474ba2cf1848c236d8cc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46103 } } peers { permanent_uuid: "888571675f4944c4b6b513528b43402a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39937 } } peers { permanent_uuid: "78d7400d33184734835817f4615ab778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38551 } }
I20260928 08:48:58.483026 20203 ts_tablet_manager.cc:1428] T e7a975c160be4b5b8526c2aec53cc6b7 P 888571675f4944c4b6b513528b43402a: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260928 08:48:58.520459 20207 raft_consensus.cc:482] T e7a975c160be4b5b8526c2aec53cc6b7 P 0d725ccac088474ba2cf1848c236d8cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:48:58.520804 20207 raft_consensus.cc:504] T e7a975c160be4b5b8526c2aec53cc6b7 P 0d725ccac088474ba2cf1848c236d8cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d725ccac088474ba2cf1848c236d8cc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46103 } } peers { permanent_uuid: "888571675f4944c4b6b513528b43402a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39937 } } peers { permanent_uuid: "78d7400d33184734835817f4615ab778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38551 } }
I20260928 08:48:58.522711 20207 leader_election.cc:288] T e7a975c160be4b5b8526c2aec53cc6b7 P 0d725ccac088474ba2cf1848c236d8cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 888571675f4944c4b6b513528b43402a (127.0.97.1:39937), 78d7400d33184734835817f4615ab778 (127.0.97.2:38551)
W20260928 08:48:58.523099 20048 tablet.cc:2369] T e7a975c160be4b5b8526c2aec53cc6b7 P 78d7400d33184734835817f4615ab778: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:48:58.529359 20184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:58.523135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:39937 (local address 127.0.97.3:59961)
0928 08:48:58.523519 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:58.523541 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:58.523680 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:58.524267 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:58.524272 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:58.524297 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:48:58.524524 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:58.524537 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:58.525943 (+ 1406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:58.525948 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:58.527381 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:58.527394 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:58.528559 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:58.528564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:58.528758 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:58.528783 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 08:48:58.528915 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 08:48:58.529057 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":1}
I20260928 08:48:58.529330 20198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:58.523218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59961 (local address 127.0.97.1:39937)
0928 08:48:58.523435 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:58.523443 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:58.523621 (+ 178us) server_negotiation.cc:407] Connection header received
0928 08:48:58.523882 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:58.523898 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:58.523981 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:58.524078 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:48:58.524712 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:58.525786 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:58.527537 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:58.528404 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:58.528544 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:58.528649 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:58.528932 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 08:48:58.529062 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 08:48:58.530171 20209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:58.523228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:38551 (local address 127.0.97.3:48293)
0928 08:48:58.524423 (+ 1195us) negotiation.cc:107] Waiting for socket to connect
0928 08:48:58.524442 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:48:58.524562 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:48:58.524987 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:48:58.524991 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:48:58.525015 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:48:58.525322 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:58.525351 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:58.526713 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:58.526718 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:58.528142 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:48:58.528154 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:48:58.529328 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:48:58.529335 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:48:58.529574 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:58.529604 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 08:48:58.529733 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 08:48:58.529844 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1071,"thread_start_us":821,"threads_started":1}
I20260928 08:48:58.530282 20197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:58.523772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:48293 (local address 127.0.97.2:38551)
0928 08:48:58.524004 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:58.524010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:58.524551 (+ 541us) server_negotiation.cc:407] Connection header received
0928 08:48:58.524713 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:58.524718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:58.524801 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:58.524914 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:48:58.525524 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:58.526563 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:58.528293 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:58.529111 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:58.529269 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:58.529366 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:58.529793 (+ 427us) server_negotiation.cc:299] Negotiation successful
0928 08:48:58.529990 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 08:48:58.530572 19868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a975c160be4b5b8526c2aec53cc6b7" candidate_uuid: "0d725ccac088474ba2cf1848c236d8cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "888571675f4944c4b6b513528b43402a" is_pre_election: true
I20260928 08:48:58.531262 19868 raft_consensus.cc:2454] T e7a975c160be4b5b8526c2aec53cc6b7 P 888571675f4944c4b6b513528b43402a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d725ccac088474ba2cf1848c236d8cc in term 0.
I20260928 08:48:58.531217 20002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a975c160be4b5b8526c2aec53cc6b7" candidate_uuid: "0d725ccac088474ba2cf1848c236d8cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78d7400d33184734835817f4615ab778" is_pre_election: true
I20260928 08:48:58.531769 20002 raft_consensus.cc:2454] T e7a975c160be4b5b8526c2aec53cc6b7 P 78d7400d33184734835817f4615ab778 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d725ccac088474ba2cf1848c236d8cc in term 0.
I20260928 08:48:58.532171 20070 leader_election.cc:302] T e7a975c160be4b5b8526c2aec53cc6b7 P 0d725ccac088474ba2cf1848c236d8cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d725ccac088474ba2cf1848c236d8cc, 888571675f4944c4b6b513528b43402a; no voters:
I20260928 08:48:58.532626 20207 raft_consensus.cc:2781] T e7a975c160be4b5b8526c2aec53cc6b7 P 0d725ccac088474ba2cf1848c236d8cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:48:58.532779 20207 raft_consensus.cc:482] T e7a975c160be4b5b8526c2aec53cc6b7 P 0d725ccac088474ba2cf1848c236d8cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:48:58.532893 20207 raft_consensus.cc:3037] T e7a975c160be4b5b8526c2aec53cc6b7 P 0d725ccac088474ba2cf1848c236d8cc [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:48:58.535187 20207 raft_consensus.cc:504] T e7a975c160be4b5b8526c2aec53cc6b7 P 0d725ccac088474ba2cf1848c236d8cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d725ccac088474ba2cf1848c236d8cc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46103 } } peers { permanent_uuid: "888571675f4944c4b6b513528b43402a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39937 } } peers { permanent_uuid: "78d7400d33184734835817f4615ab778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38551 } }
I20260928 08:48:58.535928 20207 leader_election.cc:288] T e7a975c160be4b5b8526c2aec53cc6b7 P 0d725ccac088474ba2cf1848c236d8cc [CANDIDATE]: Term 1 election: Requested vote from peers 888571675f4944c4b6b513528b43402a (127.0.97.1:39937), 78d7400d33184734835817f4615ab778 (127.0.97.2:38551)
I20260928 08:48:58.536540 20002 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a975c160be4b5b8526c2aec53cc6b7" candidate_uuid: "0d725ccac088474ba2cf1848c236d8cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78d7400d33184734835817f4615ab778"
I20260928 08:48:58.536541 19868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7a975c160be4b5b8526c2aec53cc6b7" candidate_uuid: "0d725ccac088474ba2cf1848c236d8cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "888571675f4944c4b6b513528b43402a"
I20260928 08:48:58.536790 20002 raft_consensus.cc:3037] T e7a975c160be4b5b8526c2aec53cc6b7 P 78d7400d33184734835817f4615ab778 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:48:58.536787 19868 raft_consensus.cc:3037] T e7a975c160be4b5b8526c2aec53cc6b7 P 888571675f4944c4b6b513528b43402a [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:48:58.540194 20002 raft_consensus.cc:2454] T e7a975c160be4b5b8526c2aec53cc6b7 P 78d7400d33184734835817f4615ab778 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d725ccac088474ba2cf1848c236d8cc in term 1.
I20260928 08:48:58.540194 19868 raft_consensus.cc:2454] T e7a975c160be4b5b8526c2aec53cc6b7 P 888571675f4944c4b6b513528b43402a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d725ccac088474ba2cf1848c236d8cc in term 1.
I20260928 08:48:58.540767 20070 leader_election.cc:302] T e7a975c160be4b5b8526c2aec53cc6b7 P 0d725ccac088474ba2cf1848c236d8cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0d725ccac088474ba2cf1848c236d8cc, 78d7400d33184734835817f4615ab778, 888571675f4944c4b6b513528b43402a; no voters:
I20260928 08:48:58.541148 20207 raft_consensus.cc:2781] T e7a975c160be4b5b8526c2aec53cc6b7 P 0d725ccac088474ba2cf1848c236d8cc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:48:58.541656 20207 raft_consensus.cc:686] T e7a975c160be4b5b8526c2aec53cc6b7 P 0d725ccac088474ba2cf1848c236d8cc [term 1 LEADER]: Becoming Leader. State: Replica: 0d725ccac088474ba2cf1848c236d8cc, State: Running, Role: LEADER
I20260928 08:48:58.542287 20207 consensus_queue.cc:237] T e7a975c160be4b5b8526c2aec53cc6b7 P 0d725ccac088474ba2cf1848c236d8cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d725ccac088474ba2cf1848c236d8cc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46103 } } peers { permanent_uuid: "888571675f4944c4b6b513528b43402a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39937 } } peers { permanent_uuid: "78d7400d33184734835817f4615ab778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38551 } }
I20260928 08:48:58.547523 19723 catalog_manager.cc:5495] T e7a975c160be4b5b8526c2aec53cc6b7 P 0d725ccac088474ba2cf1848c236d8cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d725ccac088474ba2cf1848c236d8cc (127.0.97.3). New cstate: current_term: 1 leader_uuid: "0d725ccac088474ba2cf1848c236d8cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d725ccac088474ba2cf1848c236d8cc" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 46103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "888571675f4944c4b6b513528b43402a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78d7400d33184734835817f4615ab778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38551 } health_report { overall_health: UNKNOWN } } }
W20260928 08:48:58.581843 20182 tablet.cc:2369] T e7a975c160be4b5b8526c2aec53cc6b7 P 0d725ccac088474ba2cf1848c236d8cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:48:58.593740 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:58.587983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43560 (local address 127.0.97.3:46103)
0928 08:48:58.588178 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:58.588186 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:58.588276 (+ 90us) server_negotiation.cc:407] Connection header received
0928 08:48:58.588524 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:58.588527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:58.588616 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:58.588768 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:48:58.589150 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:58.589949 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:58.591465 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:58.591893 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:58.592033 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:58.592575 (+ 542us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:48:58.593117 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:48:58.593281 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:58.593364 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 08:48:58.593462 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 08:48:58.603984 19868 raft_consensus.cc:1261] T e7a975c160be4b5b8526c2aec53cc6b7 P 888571675f4944c4b6b513528b43402a [term 1 FOLLOWER]: Refusing update from remote peer 0d725ccac088474ba2cf1848c236d8cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:48:58.605424 20207 consensus_queue.cc:1035] T e7a975c160be4b5b8526c2aec53cc6b7 P 0d725ccac088474ba2cf1848c236d8cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "888571675f4944c4b6b513528b43402a" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 39937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:48:58.605506 20002 raft_consensus.cc:1261] T e7a975c160be4b5b8526c2aec53cc6b7 P 78d7400d33184734835817f4615ab778 [term 1 FOLLOWER]: Refusing update from remote peer 0d725ccac088474ba2cf1848c236d8cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:48:58.606796 20207 consensus_queue.cc:1035] T e7a975c160be4b5b8526c2aec53cc6b7 P 0d725ccac088474ba2cf1848c236d8cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "78d7400d33184734835817f4615ab778" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 38551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:48:58.624801 20213 mvcc.cc:205] Tried to move back new op lower bound from 7334237546905047040 to 7334237546673508352. Current Snapshot: MvccSnapshot[applied={T|T < 7334237546905047040}]
I20260928 08:48:58.629309 20214 mvcc.cc:205] Tried to move back new op lower bound from 7334237546905047040 to 7334237546673508352. Current Snapshot: MvccSnapshot[applied={T|T < 7334237546905047040}]
I20260928 08:48:58.642691 19724 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:48:58.643040 19724 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:48:58.647682 19724 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7d749479d2954260820ade10cd2ee5e4: Sending DeleteTablet for 3 replicas of tablet e7a975c160be4b5b8526c2aec53cc6b7
I20260928 08:48:58.649536 19848 tablet_service.cc:1514] Processing DeleteTablet for tablet e7a975c160be4b5b8526c2aec53cc6b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:48:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:33286
I20260928 08:48:58.650290 20116 tablet_service.cc:1514] Processing DeleteTablet for tablet e7a975c160be4b5b8526c2aec53cc6b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:48:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:43550
I20260928 08:48:58.650704 19982 tablet_service.cc:1514] Processing DeleteTablet for tablet e7a975c160be4b5b8526c2aec53cc6b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:48:58 UTC) from {username='kudu', principal='kudu/127.0.97.62@KRBTEST.COM'} at 127.0.0.1:42480
I20260928 08:48:58.651608 20225 tablet_replica.cc:330] T e7a975c160be4b5b8526c2aec53cc6b7 P 78d7400d33184734835817f4615ab778: stopping tablet replica
I20260928 08:48:58.651899 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 19785
I20260928 08:48:58.652328 20225 raft_consensus.cc:2229] T e7a975c160be4b5b8526c2aec53cc6b7 P 78d7400d33184734835817f4615ab778 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:48:58.652942 20225 raft_consensus.cc:2258] T e7a975c160be4b5b8526c2aec53cc6b7 P 78d7400d33184734835817f4615ab778 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:48:58.655324 20225 ts_tablet_manager.cc:1905] T e7a975c160be4b5b8526c2aec53cc6b7 P 78d7400d33184734835817f4615ab778: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:48:58.663156 20225 ts_tablet_manager.cc:1918] T e7a975c160be4b5b8526c2aec53cc6b7 P 78d7400d33184734835817f4615ab778: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:48:58.663486 20225 log.cc:1196] T e7a975c160be4b5b8526c2aec53cc6b7 P 78d7400d33184734835817f4615ab778: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-1/wal/wals/e7a975c160be4b5b8526c2aec53cc6b7
I20260928 08:48:58.664198 20225 ts_tablet_manager.cc:1939] T e7a975c160be4b5b8526c2aec53cc6b7 P 78d7400d33184734835817f4615ab778: Deleting consensus metadata
I20260928 08:48:58.666371 19708 catalog_manager.cc:4841] TS 78d7400d33184734835817f4615ab778 (127.0.97.2:38551): tablet e7a975c160be4b5b8526c2aec53cc6b7 (table test-table [id=f4f2aef01a2e461f9fb24bbdbe0a509a]) successfully deleted
I20260928 08:48:58.669219 20224 tablet_replica.cc:330] T e7a975c160be4b5b8526c2aec53cc6b7 P 0d725ccac088474ba2cf1848c236d8cc: stopping tablet replica
I20260928 08:48:58.669721 20224 raft_consensus.cc:2229] T e7a975c160be4b5b8526c2aec53cc6b7 P 0d725ccac088474ba2cf1848c236d8cc [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:48:58.670413 20224 raft_consensus.cc:2258] T e7a975c160be4b5b8526c2aec53cc6b7 P 0d725ccac088474ba2cf1848c236d8cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:48:58.672856 20224 ts_tablet_manager.cc:1905] T e7a975c160be4b5b8526c2aec53cc6b7 P 0d725ccac088474ba2cf1848c236d8cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:48:58.674862 20223 tablet_replica.cc:330] T e7a975c160be4b5b8526c2aec53cc6b7 P 888571675f4944c4b6b513528b43402a: stopping tablet replica
I20260928 08:48:58.675441 20223 raft_consensus.cc:2229] T e7a975c160be4b5b8526c2aec53cc6b7 P 888571675f4944c4b6b513528b43402a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:48:58.676095 20223 raft_consensus.cc:2258] T e7a975c160be4b5b8526c2aec53cc6b7 P 888571675f4944c4b6b513528b43402a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:48:58.681017 20224 ts_tablet_manager.cc:1918] T e7a975c160be4b5b8526c2aec53cc6b7 P 0d725ccac088474ba2cf1848c236d8cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:48:58.681221 20224 log.cc:1196] T e7a975c160be4b5b8526c2aec53cc6b7 P 0d725ccac088474ba2cf1848c236d8cc: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-2/wal/wals/e7a975c160be4b5b8526c2aec53cc6b7
I20260928 08:48:58.682024 20224 ts_tablet_manager.cc:1939] T e7a975c160be4b5b8526c2aec53cc6b7 P 0d725ccac088474ba2cf1848c236d8cc: Deleting consensus metadata
I20260928 08:48:58.682690 20223 ts_tablet_manager.cc:1905] T e7a975c160be4b5b8526c2aec53cc6b7 P 888571675f4944c4b6b513528b43402a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:48:58.684012 19709 catalog_manager.cc:4841] TS 0d725ccac088474ba2cf1848c236d8cc (127.0.97.3:46103): tablet e7a975c160be4b5b8526c2aec53cc6b7 (table test-table [id=f4f2aef01a2e461f9fb24bbdbe0a509a]) successfully deleted
Sep 28 08:48:58 dist-test-slave-n6xs krb5kdc[19679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.1@KRBTEST.COM
I20260928 08:48:58.695479 20223 ts_tablet_manager.cc:1918] T e7a975c160be4b5b8526c2aec53cc6b7 P 888571675f4944c4b6b513528b43402a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:48:58.695684 20223 log.cc:1196] T e7a975c160be4b5b8526c2aec53cc6b7 P 888571675f4944c4b6b513528b43402a: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790585022442112-388-0/minicluster-data/ts-0/wal/wals/e7a975c160be4b5b8526c2aec53cc6b7
I20260928 08:48:58.696285 20223 ts_tablet_manager.cc:1939] T e7a975c160be4b5b8526c2aec53cc6b7 P 888571675f4944c4b6b513528b43402a: Deleting consensus metadata
I20260928 08:48:58.698314 19709 catalog_manager.cc:4841] TS 888571675f4944c4b6b513528b43402a (127.0.97.1:39937): tablet e7a975c160be4b5b8526c2aec53cc6b7 (table test-table [id=f4f2aef01a2e461f9fb24bbdbe0a509a]) successfully deleted
I20260928 08:48:58.700471 20198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:58.654197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33296 (local address 127.0.97.1:39937)
0928 08:48:58.654440 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:58.654449 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:58.671206 (+ 16757us) server_negotiation.cc:407] Connection header received
0928 08:48:58.671782 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:58.671791 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:58.672061 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:58.672158 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:58.676610 (+ 4452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:58.677940 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:58.685875 (+ 7935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:58.686718 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:58.686889 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:58.692674 (+ 5785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:58.692692 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:58.692712 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:48:58.692771 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:58.697207 (+ 4436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:58.698175 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:58.698186 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:58.698190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:58.698333 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:58.698788 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:58.698794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:58.698797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:58.699173 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:58.699504 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:58.699883 (+ 379us) server_negotiation.cc:299] Negotiation successful
0928 08:48:58.700121 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 08:48:58.701218 19908 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:48:58.792408 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 19785
I20260928 08:48:58.811506 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 19919
Sep 28 08:48:58 dist-test-slave-n6xs krb5kdc[19679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.2@KRBTEST.COM
I20260928 08:48:58.828068 20197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:58.812496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42484 (local address 127.0.97.2:38551)
0928 08:48:58.812677 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:58.812683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:58.812705 (+ 22us) server_negotiation.cc:407] Connection header received
0928 08:48:58.812778 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:58.812783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:58.812871 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:58.812981 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:58.814227 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:58.815005 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:58.816399 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:58.816815 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:58.816948 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:58.821892 (+ 4944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:58.821911 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:58.821927 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:48:58.821972 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:58.825050 (+ 3078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:58.826023 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:58.826030 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:58.826032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:58.826157 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:58.826557 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:58.826560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:58.826562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:58.826943 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:58.827163 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:58.827507 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 08:48:58.827673 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 08:48:58.828784 20042 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:48:58.924727 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 19919
I20260928 08:48:58.942574 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 20053
Sep 28 08:48:58 dist-test-slave-n6xs krb5kdc[19679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585335, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.3@KRBTEST.COM
I20260928 08:48:58.959856 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:58.943478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43570 (local address 127.0.97.3:46103)
0928 08:48:58.943720 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:58.943726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:58.943748 (+ 22us) server_negotiation.cc:407] Connection header received
0928 08:48:58.944133 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:58.944138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:58.944212 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:58.944326 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:58.945619 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:58.946446 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:58.947758 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:58.948313 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:58.948465 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:58.953471 (+ 5006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:58.953487 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:58.953504 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:48:58.953555 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:58.956758 (+ 3203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:58.957710 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:58.957717 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:58.957719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:58.957879 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:48:58.958306 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:48:58.958309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:48:58.958311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:48:58.958684 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:58.958922 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:58.959326 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 08:48:58.959491 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260928 08:48:58.960618 20176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:48:59.087896 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 20053
I20260928 08:48:59.107697 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 19692
I20260928 08:48:59.108780 19754 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:48:59.222093 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 19692
2026-09-28T08:48:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4597 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T08:48:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:48:59Z Disabled control of system clock
I20260928 08:48:59.301410 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40525
--webserver_interface=127.0.97.62
--webserver_port=0
--builtin_ntp_servers=127.0.97.20:33321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 08:48:59.652307 20234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:48:59.652638 20234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:48:59.652735 20234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:48:59.660235 20234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:48:59.660362 20234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:48:59.660400 20234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:48:59.660460 20234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:48:59.660506 20234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:48:59.670137 20234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40525
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:48:59.671427 20234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:48:59.673661 20234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:48:59.685968 20242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:59.685909 20240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:48:59.686054 20239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:48:59.687120 20234 server_base.cc:1034] running on GCE node
I20260928 08:48:59.688107 20234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:48:59.689635 20234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:48:59.690871 20234 hybrid_clock.cc:648] HybridClock initialized: now 1790585339690808 us; error 76 us; skew 500 ppm
I20260928 08:48:59.691259 20234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:48:59.699318 20234 webserver.cc:460] Webserver started at http://127.0.97.62:46231/ using document root <none> and password file <none>
I20260928 08:48:59.700242 20234 fs_manager.cc:359] Metadata directory not provided
I20260928 08:48:59.700382 20234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:48:59.700717 20234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:48:59.703872 20234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "7ae0259e51214be486f8af5608b73a79"
format_stamp: "Formatted at 2026-09-28 08:48:59 on dist-test-slave-n6xs"
server_key: "e1b710a85a8fee1e478f9c1cc8a0e6ad"
server_key_iv: "8a8e746d3e3508e0eea7e6d5d476432a"
server_key_version: "encryptionkey@0"
I20260928 08:48:59.704741 20234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "7ae0259e51214be486f8af5608b73a79"
format_stamp: "Formatted at 2026-09-28 08:48:59 on dist-test-slave-n6xs"
server_key: "e1b710a85a8fee1e478f9c1cc8a0e6ad"
server_key_iv: "8a8e746d3e3508e0eea7e6d5d476432a"
server_key_version: "encryptionkey@0"
I20260928 08:48:59.710332 20234 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 08:48:59.714080 20248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:48:59.715499 20234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 08:48:59.715708 20234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "7ae0259e51214be486f8af5608b73a79"
format_stamp: "Formatted at 2026-09-28 08:48:59 on dist-test-slave-n6xs"
server_key: "e1b710a85a8fee1e478f9c1cc8a0e6ad"
server_key_iv: "8a8e746d3e3508e0eea7e6d5d476432a"
server_key_version: "encryptionkey@0"
I20260928 08:48:59.716042 20234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:48:59.739576 20234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:48:59.740661 20234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:48:59.740986 20234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:48:59.761022 20234 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40525
I20260928 08:48:59.761047 20299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40525 every 8 connection(s)
I20260928 08:48:59.762476 20234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:48:59.766731 20300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:48:59.767769 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 20234
I20260928 08:48:59.768002 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:48:59.768607 388 external_mini_cluster.cc:1428] Setting key cb9d3a8270a5c4346da5b636e28acc87
I20260928 08:48:59.781129 20300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ae0259e51214be486f8af5608b73a79: Bootstrap starting.
I20260928 08:48:59.783401 20303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:48:59.770797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41080 (local address 127.0.97.62:40525)
0928 08:48:59.771998 (+ 1201us) server_negotiation.cc:206] Beginning negotiation
0928 08:48:59.772014 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 08:48:59.772058 (+ 44us) server_negotiation.cc:407] Connection header received
0928 08:48:59.773982 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:48:59.774009 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:48:59.774601 (+ 592us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:48:59.775036 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:48:59.775655 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:59.777080 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:59.778519 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:48:59.779248 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:48:59.779475 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:48:59.780827 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:48:59.780839 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:48:59.780855 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:48:59.781504 (+ 649us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:48:59.781591 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:48:59.781614 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:48:59.781748 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 08:48:59.782042 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 08:48:59.782381 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":426,"threads_started":1}
I20260928 08:48:59.784713 20300 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ae0259e51214be486f8af5608b73a79: Neither blocks nor log segments found. Creating new log.
I20260928 08:48:59.785809 20300 log.cc:824] T 00000000000000000000000000000000 P 7ae0259e51214be486f8af5608b73a79: Log is configured to *not* fsync() on all Append() calls
I20260928 08:48:59.788707 20300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ae0259e51214be486f8af5608b73a79: No bootstrap required, opened a new log
I20260928 08:48:59.796536 20300 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ae0259e51214be486f8af5608b73a79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ae0259e51214be486f8af5608b73a79" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40525 } }
I20260928 08:48:59.797047 20300 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ae0259e51214be486f8af5608b73a79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:48:59.797166 20300 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ae0259e51214be486f8af5608b73a79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ae0259e51214be486f8af5608b73a79, State: Initialized, Role: FOLLOWER
I20260928 08:48:59.797847 20300 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ae0259e51214be486f8af5608b73a79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ae0259e51214be486f8af5608b73a79" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40525 } }
I20260928 08:48:59.798069 20300 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ae0259e51214be486f8af5608b73a79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:48:59.798221 20300 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ae0259e51214be486f8af5608b73a79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:48:59.798403 20300 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ae0259e51214be486f8af5608b73a79 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:48:59.800802 20300 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ae0259e51214be486f8af5608b73a79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ae0259e51214be486f8af5608b73a79" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40525 } }
I20260928 08:48:59.801363 20300 leader_election.cc:302] T 00000000000000000000000000000000 P 7ae0259e51214be486f8af5608b73a79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ae0259e51214be486f8af5608b73a79; no voters:
I20260928 08:48:59.802002 20300 leader_election.cc:288] T 00000000000000000000000000000000 P 7ae0259e51214be486f8af5608b73a79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:48:59.802183 20305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ae0259e51214be486f8af5608b73a79 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:48:59.802907 20305 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ae0259e51214be486f8af5608b73a79 [term 1 LEADER]: Becoming Leader. State: Replica: 7ae0259e51214be486f8af5608b73a79, State: Running, Role: LEADER
I20260928 08:48:59.803500 20305 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ae0259e51214be486f8af5608b73a79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ae0259e51214be486f8af5608b73a79" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40525 } }
I20260928 08:48:59.804270 20300 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ae0259e51214be486f8af5608b73a79 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:48:59.806696 20306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ae0259e51214be486f8af5608b73a79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ae0259e51214be486f8af5608b73a79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ae0259e51214be486f8af5608b73a79" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40525 } } }
I20260928 08:48:59.806787 20307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ae0259e51214be486f8af5608b73a79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ae0259e51214be486f8af5608b73a79. Latest consensus state: current_term: 1 leader_uuid: "7ae0259e51214be486f8af5608b73a79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ae0259e51214be486f8af5608b73a79" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40525 } } }
I20260928 08:48:59.807230 20307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ae0259e51214be486f8af5608b73a79 [sys.catalog]: This master's current role is: LEADER
I20260928 08:48:59.807226 20306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ae0259e51214be486f8af5608b73a79 [sys.catalog]: This master's current role is: LEADER
I20260928 08:48:59.808703 20313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:48:59.817534 20313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:48:59.827733 20313 catalog_manager.cc:1347] Generated new cluster ID: e2d7977cd858494cb02502a89003147a
I20260928 08:48:59.827910 20313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:48:59.882735 20313 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:48:59.884474 20313 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:48:59.896653 20313 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ae0259e51214be486f8af5608b73a79: Generated new TSK 0
I20260928 08:48:59.897716 20313 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:48:59.904465 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.1:0
--local_ip_for_outbound_sockets=127.0.97.1
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40525
--builtin_ntp_servers=127.0.97.20:33321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 08:49:00.256168 20324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:49:00.256546 20324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:49:00.256701 20324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:49:00.264204 20324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:49:00.264364 20324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:49:00.264526 20324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.1
I20260928 08:49:00.273939 20324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.97.1
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.1
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:49:00.275353 20324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:49:00.277292 20324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:49:00.291105 20329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:00.291174 20332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:00.291174 20330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:49:00.292107 20324 server_base.cc:1034] running on GCE node
I20260928 08:49:00.292826 20324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:49:00.293864 20324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:49:00.295135 20324 hybrid_clock.cc:648] HybridClock initialized: now 1790585340295080 us; error 50 us; skew 500 ppm
I20260928 08:49:00.295509 20324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:49:00.298099 20324 webserver.cc:460] Webserver started at http://127.0.97.1:42053/ using document root <none> and password file <none>
I20260928 08:49:00.298869 20324 fs_manager.cc:359] Metadata directory not provided
I20260928 08:49:00.299012 20324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:49:00.299343 20324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:49:00.302403 20324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-0/data/instance:
uuid: "399a25258b634c9696bb5c394ab07c46"
format_stamp: "Formatted at 2026-09-28 08:49:00 on dist-test-slave-n6xs"
server_key: "8a67385d0a2dab3568d4e82932fadcfd"
server_key_iv: "9e1f858c80e200d36a8df1ab8478076b"
server_key_version: "encryptionkey@0"
I20260928 08:49:00.303246 20324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-0/wal/instance:
uuid: "399a25258b634c9696bb5c394ab07c46"
format_stamp: "Formatted at 2026-09-28 08:49:00 on dist-test-slave-n6xs"
server_key: "8a67385d0a2dab3568d4e82932fadcfd"
server_key_iv: "9e1f858c80e200d36a8df1ab8478076b"
server_key_version: "encryptionkey@0"
I20260928 08:49:00.309187 20324 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 08:49:00.312937 20338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:49:00.314393 20324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 08:49:00.314587 20324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-0/wal
uuid: "399a25258b634c9696bb5c394ab07c46"
format_stamp: "Formatted at 2026-09-28 08:49:00 on dist-test-slave-n6xs"
server_key: "8a67385d0a2dab3568d4e82932fadcfd"
server_key_iv: "9e1f858c80e200d36a8df1ab8478076b"
server_key_version: "encryptionkey@0"
I20260928 08:49:00.314841 20324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:49:00.351414 20324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:49:00.352540 20324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:49:00.352890 20324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:49:00.354146 20324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:49:00.355904 20324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:49:00.356045 20324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:49:00.356190 20324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:49:00.356295 20324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:49:00.393544 20324 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.1:44639
I20260928 08:49:00.393608 20450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.1:44639 every 8 connection(s)
I20260928 08:49:00.395057 20324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-0/data/info.pb
I20260928 08:49:00.404433 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 20324
I20260928 08:49:00.404726 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-0/wal/instance
I20260928 08:49:00.405275 388 external_mini_cluster.cc:1428] Setting key a04d12772007811f42fec20318d0f6d7
I20260928 08:49:00.408541 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.2:0
--local_ip_for_outbound_sockets=127.0.97.2
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40525
--builtin_ntp_servers=127.0.97.20:33321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 08:49:00.409966 20453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:00.397925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:37635 (local address 127.0.97.62:40525)
0928 08:49:00.398404 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:00.398409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:00.399987 (+ 1578us) server_negotiation.cc:407] Connection header received
0928 08:49:00.401902 (+ 1915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:00.401908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:00.402009 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:00.402132 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:00.403290 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:00.404142 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:00.406255 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:00.406863 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:00.406996 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:00.408887 (+ 1891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:00.408896 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:00.408903 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:49:00.408929 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:00.408981 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:49:00.409000 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:00.409094 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:00.409374 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 08:49:00.409494 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":226,"threads_started":1}
I20260928 08:49:00.410943 20454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:00.398576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40525 (local address 127.0.97.1:37635)
0928 08:49:00.399740 (+ 1164us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:00.399778 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 08:49:00.401592 (+ 1814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:49:00.402389 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:49:00.402397 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:49:00.402495 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:49:00.403111 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:00.403129 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:00.404319 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:00.404323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:00.406103 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:00.406116 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:00.407259 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:00.407266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:00.407576 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:00.408412 (+ 836us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:49:00.408418 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:49:00.408476 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:49:00.408628 (+ 152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:49:00.409108 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:49:00.409118 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:49:00.409124 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 08:49:00.409325 (+ 201us) client_negotiation.cc:240] Negotiation successful
0928 08:49:00.409715 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"thread_start_us":497,"threads_started":1}
I20260928 08:49:00.412609 20451 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40525
I20260928 08:49:00.412948 20451 heartbeater.cc:464] Registering TS with master...
I20260928 08:49:00.413908 20451 heartbeater.cc:511] Master 127.0.97.62:40525 requested a full tablet report, sending...
I20260928 08:49:00.416389 20265 ts_manager.cc:194] Registered new tserver with Master: 399a25258b634c9696bb5c394ab07c46 (127.0.97.1:44639)
I20260928 08:49:00.417897 20265 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.1:37635
W20260928 08:49:00.756983 20455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:49:00.757299 20455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:49:00.757481 20455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:49:00.765200 20455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:49:00.765379 20455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:49:00.765573 20455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.2
I20260928 08:49:00.775254 20455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.97.2
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.2
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:49:00.776785 20455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:49:00.778676 20455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:49:00.793088 20460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:00.793308 20461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:00.793351 20463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:49:00.794649 20455 server_base.cc:1034] running on GCE node
I20260928 08:49:00.795346 20455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:49:00.796445 20455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:49:00.797652 20455 hybrid_clock.cc:648] HybridClock initialized: now 1790585340797603 us; error 38 us; skew 500 ppm
I20260928 08:49:00.797988 20455 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:49:00.800520 20455 webserver.cc:460] Webserver started at http://127.0.97.2:39263/ using document root <none> and password file <none>
I20260928 08:49:00.801265 20455 fs_manager.cc:359] Metadata directory not provided
I20260928 08:49:00.801390 20455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:49:00.801726 20455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:49:00.804924 20455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-1/data/instance:
uuid: "a753ff8fb0424e418de13498aac8da33"
format_stamp: "Formatted at 2026-09-28 08:49:00 on dist-test-slave-n6xs"
server_key: "582ef5d29bf59220665a48658f17555a"
server_key_iv: "7c0bdb26d89382128fa34416b428f068"
server_key_version: "encryptionkey@0"
I20260928 08:49:00.805855 20455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-1/wal/instance:
uuid: "a753ff8fb0424e418de13498aac8da33"
format_stamp: "Formatted at 2026-09-28 08:49:00 on dist-test-slave-n6xs"
server_key: "582ef5d29bf59220665a48658f17555a"
server_key_iv: "7c0bdb26d89382128fa34416b428f068"
server_key_version: "encryptionkey@0"
I20260928 08:49:00.811385 20455 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:49:00.815299 20469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:49:00.816906 20455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:49:00.817093 20455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-1/wal
uuid: "a753ff8fb0424e418de13498aac8da33"
format_stamp: "Formatted at 2026-09-28 08:49:00 on dist-test-slave-n6xs"
server_key: "582ef5d29bf59220665a48658f17555a"
server_key_iv: "7c0bdb26d89382128fa34416b428f068"
server_key_version: "encryptionkey@0"
I20260928 08:49:00.817340 20455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:49:00.846748 20455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:49:00.847743 20455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:49:00.848063 20455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:49:00.849179 20455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:49:00.850780 20455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:49:00.850865 20455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:49:00.851022 20455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:49:00.851102 20455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:49:00.885905 20455 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.2:46417
I20260928 08:49:00.885959 20581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.2:46417 every 8 connection(s)
I20260928 08:49:00.887348 20455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-1/data/info.pb
I20260928 08:49:00.888044 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 20455
I20260928 08:49:00.888219 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-1/wal/instance
I20260928 08:49:00.888628 388 external_mini_cluster.cc:1428] Setting key 7204dff8b1dfb80a4c70624fa53d7f70
I20260928 08:49:00.891885 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.97.3:0
--local_ip_for_outbound_sockets=127.0.97.3
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40525
--builtin_ntp_servers=127.0.97.20:33321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 08:49:00.902067 20453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:00.890505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.2:40261 (local address 127.0.97.62:40525)
0928 08:49:00.890812 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:00.890818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:00.892467 (+ 1649us) server_negotiation.cc:407] Connection header received
0928 08:49:00.894351 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:00.894359 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:00.894456 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:00.894583 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:00.895722 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:00.897066 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:00.898640 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:00.899380 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:00.899484 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:00.901266 (+ 1782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:00.901271 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:00.901274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:49:00.901296 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:00.901339 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:49:00.901351 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:00.901425 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:00.901750 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 08:49:00.901835 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190}
I20260928 08:49:00.903900 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:00.891170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40525 (local address 127.0.97.2:40261)
0928 08:49:00.892279 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:00.892330 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 08:49:00.894053 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:49:00.894787 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:49:00.894796 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:49:00.894890 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:49:00.895516 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:00.895539 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:00.897206 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:00.897210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:00.898483 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:00.898494 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:00.899620 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:00.899630 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:00.900020 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:00.900889 (+ 869us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:49:00.900894 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:49:00.900955 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:49:00.901064 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:49:00.901481 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:49:00.901497 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:49:00.901510 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 08:49:00.901749 (+ 239us) client_negotiation.cc:240] Negotiation successful
0928 08:49:00.902241 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":341,"threads_started":1}
I20260928 08:49:00.905768 20582 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40525
I20260928 08:49:00.906118 20582 heartbeater.cc:464] Registering TS with master...
I20260928 08:49:00.906980 20582 heartbeater.cc:511] Master 127.0.97.62:40525 requested a full tablet report, sending...
I20260928 08:49:00.908622 20265 ts_manager.cc:194] Registered new tserver with Master: a753ff8fb0424e418de13498aac8da33 (127.0.97.2:46417)
I20260928 08:49:00.909543 20265 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.2:40261
W20260928 08:49:01.239602 20585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:49:01.240072 20585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:49:01.240237 20585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:49:01.247946 20585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:49:01.248113 20585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:49:01.248265 20585 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.97.3
I20260928 08:49:01.259127 20585 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:33321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.97.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.97.3
--webserver_port=0
--tserver_master_addrs=127.0.97.62:40525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.97.3
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:49:01.260673 20585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:49:01.262492 20585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:49:01.276135 20593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:01.276268 20590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:01.276198 20591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:49:01.277236 20585 server_base.cc:1034] running on GCE node
I20260928 08:49:01.277912 20585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:49:01.278973 20585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:49:01.280200 20585 hybrid_clock.cc:648] HybridClock initialized: now 1790585341280146 us; error 54 us; skew 500 ppm
I20260928 08:49:01.280539 20585 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 08:49:01.282918 20585 webserver.cc:460] Webserver started at http://127.0.97.3:46041/ using document root <none> and password file <none>
I20260928 08:49:01.283681 20585 fs_manager.cc:359] Metadata directory not provided
I20260928 08:49:01.283813 20585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:49:01.284192 20585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:49:01.287282 20585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-2/data/instance:
uuid: "a7a5af83c69e4f91b3ff5294ead7d4f3"
format_stamp: "Formatted at 2026-09-28 08:49:01 on dist-test-slave-n6xs"
server_key: "e5a101506c059915864215dce5e57e10"
server_key_iv: "dd77111db5b3530794d27b8d7091405f"
server_key_version: "encryptionkey@0"
I20260928 08:49:01.288251 20585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-2/wal/instance:
uuid: "a7a5af83c69e4f91b3ff5294ead7d4f3"
format_stamp: "Formatted at 2026-09-28 08:49:01 on dist-test-slave-n6xs"
server_key: "e5a101506c059915864215dce5e57e10"
server_key_iv: "dd77111db5b3530794d27b8d7091405f"
server_key_version: "encryptionkey@0"
I20260928 08:49:01.293977 20585 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:49:01.297925 20599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:49:01.299489 20585 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:49:01.299690 20585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-2/wal
uuid: "a7a5af83c69e4f91b3ff5294ead7d4f3"
format_stamp: "Formatted at 2026-09-28 08:49:01 on dist-test-slave-n6xs"
server_key: "e5a101506c059915864215dce5e57e10"
server_key_iv: "dd77111db5b3530794d27b8d7091405f"
server_key_version: "encryptionkey@0"
I20260928 08:49:01.299989 20585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:49:01.316161 20585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:49:01.317121 20585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:49:01.317422 20585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:49:01.318565 20585 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 08:49:01.320426 20585 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 08:49:01.320541 20585 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 08:49:01.320677 20585 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 08:49:01.320760 20585 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 08:49:01.354800 20585 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.3:42171
I20260928 08:49:01.354871 20711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.3:42171 every 8 connection(s)
I20260928 08:49:01.356292 20585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-2/data/info.pb
I20260928 08:49:01.362444 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 20585
I20260928 08:49:01.362622 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-2/wal/instance
I20260928 08:49:01.363135 388 external_mini_cluster.cc:1428] Setting key cf8b2b7a462fb33fac683ff6cfcf543a
I20260928 08:49:01.370839 20453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:01.359225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:36313 (local address 127.0.97.62:40525)
0928 08:49:01.359426 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:01.359431 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:01.361110 (+ 1679us) server_negotiation.cc:407] Connection header received
0928 08:49:01.363523 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:01.363528 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:01.363618 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:01.363769 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:01.365091 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.365891 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.367673 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.368218 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.368360 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:01.369906 (+ 1546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:01.369911 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:01.369914 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:49:01.369936 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:01.369971 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:49:01.369983 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:01.370091 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:01.370355 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 08:49:01.370472 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 08:49:01.372067 20714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:01.359917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40525 (local address 127.0.97.3:36313)
0928 08:49:01.360917 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:01.360962 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 08:49:01.363133 (+ 2171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:49:01.364084 (+ 951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:49:01.364115 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:49:01.364252 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:49:01.364916 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:01.364935 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.366088 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.366092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:01.367499 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:01.367512 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.368470 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.368478 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:01.368721 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:01.369469 (+ 748us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 08:49:01.369472 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:49:01.369523 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 08:49:01.369614 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:49:01.370091 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:49:01.370106 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:49:01.370117 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 08:49:01.370330 (+ 213us) client_negotiation.cc:240] Negotiation successful
0928 08:49:01.370789 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":312,"threads_started":1}
I20260928 08:49:01.373812 20712 heartbeater.cc:347] Connected to a master server at 127.0.97.62:40525
I20260928 08:49:01.374140 20712 heartbeater.cc:464] Registering TS with master...
I20260928 08:49:01.374936 20712 heartbeater.cc:511] Master 127.0.97.62:40525 requested a full tablet report, sending...
I20260928 08:49:01.376593 20265 ts_manager.cc:194] Registered new tserver with Master: a7a5af83c69e4f91b3ff5294ead7d4f3 (127.0.97.3:42171)
I20260928 08:49:01.377739 20265 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.97.3:36313
I20260928 08:49:01.379333 388 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 08:49:01.390000 20453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:01.383969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41090 (local address 127.0.97.62:40525)
0928 08:49:01.384154 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:01.384159 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:01.384562 (+ 403us) server_negotiation.cc:407] Connection header received
0928 08:49:01.384778 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:01.384783 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:01.384873 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:01.384981 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:01.385461 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.386162 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.387610 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.388120 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.388220 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:01.389366 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:01.389371 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:01.389374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:49:01.389408 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:01.389441 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:49:01.389454 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:01.389536 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:01.389655 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 08:49:01.389741 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 08:49:01.394426 20265 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41090:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 08:49:01.401633 20265 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 08:49:01.422164 20451 heartbeater.cc:503] Master 127.0.97.62:40525 was elected leader, sending a full tablet report...
I20260928 08:49:01.439781 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:01.419755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57014 (local address 127.0.97.2:46417)
0928 08:49:01.430981 (+ 11226us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:01.430994 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:01.431032 (+ 38us) server_negotiation.cc:407] Connection header received
0928 08:49:01.431161 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:01.431171 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:01.431508 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:01.431730 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:49:01.432628 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.433895 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.436754 (+ 2859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.438283 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.438465 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:01.438764 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:01.439229 (+ 465us) server_negotiation.cc:299] Negotiation successful
0928 08:49:01.439449 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11036,"thread_start_us":12176,"threads_started":1}
I20260928 08:49:01.439893 20724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:01.417029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46417 (local address 127.0.0.1:57014)
0928 08:49:01.417846 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:01.417891 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 08:49:01.418068 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:49:01.431881 (+ 13813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:49:01.431893 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:49:01.431943 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:49:01.432419 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:01.432436 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.434066 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.434072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:01.436431 (+ 2359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:01.436452 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.438520 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.438527 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:01.438843 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:01.438979 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 08:49:01.439144 (+ 165us) client_negotiation.cc:240] Negotiation successful
0928 08:49:01.439390 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":38,"thread_start_us":381,"threads_started":1}
I20260928 08:49:01.443449 20726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:01.418269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44639 (local address 127.0.0.1:54246)
0928 08:49:01.435158 (+ 16889us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:01.435177 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 08:49:01.435398 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:49:01.436280 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:49:01.436285 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:49:01.436316 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:49:01.436672 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:01.436679 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.438187 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.438195 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:01.440624 (+ 2429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:01.440639 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.442612 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.442616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:01.442735 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:01.442760 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 08:49:01.442849 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 08:49:01.443018 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16739,"mutex_wait_us":65,"thread_start_us":16575,"threads_started":1}
I20260928 08:49:01.443534 20728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:01.419321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54246 (local address 127.0.97.1:44639)
0928 08:49:01.420482 (+ 1161us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:01.420551 (+ 69us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:01.435299 (+ 14748us) server_negotiation.cc:407] Connection header received
0928 08:49:01.435623 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:01.435632 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:01.435983 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:01.436201 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:49:01.436822 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.438035 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.440883 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.442417 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.442564 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:01.442907 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:01.442990 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 08:49:01.443172 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":982,"thread_start_us":862,"threads_started":1}
I20260928 08:49:01.443889 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:01.418723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46688 (local address 127.0.97.3:42171)
0928 08:49:01.433201 (+ 14478us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:01.433505 (+ 304us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:01.433564 (+ 59us) server_negotiation.cc:407] Connection header received
0928 08:49:01.433695 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:01.433708 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:01.434089 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:01.434347 (+ 258us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:49:01.436299 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.438031 (+ 1732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.441343 (+ 3312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.442741 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.442892 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:01.443140 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:01.443300 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 08:49:01.443464 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14238,"thread_start_us":14064,"threads_started":1}
I20260928 08:49:01.444128 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:01.417544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42171 (local address 127.0.0.1:46688)
0928 08:49:01.430965 (+ 13421us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:01.430990 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 08:49:01.431152 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:49:01.435583 (+ 4431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:49:01.435588 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:49:01.435610 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:49:01.436074 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:01.436090 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.438215 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.438223 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:01.441138 (+ 2915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:01.441150 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.442943 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.442949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:01.443122 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:01.443155 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 08:49:01.443232 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 08:49:01.443409 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13250,"mutex_wait_us":304,"thread_start_us":13385,"threads_started":1}
I20260928 08:49:01.447563 20517 tablet_service.cc:1467] Processing CreateTablet for tablet a6524b39b5724f048bd69d8d50745b99 (DEFAULT_TABLE table=test-table [id=3d5682e6868241d8904c217809d0ee52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:49:01.447688 20386 tablet_service.cc:1467] Processing CreateTablet for tablet a6524b39b5724f048bd69d8d50745b99 (DEFAULT_TABLE table=test-table [id=3d5682e6868241d8904c217809d0ee52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:49:01.447690 20647 tablet_service.cc:1467] Processing CreateTablet for tablet a6524b39b5724f048bd69d8d50745b99 (DEFAULT_TABLE table=test-table [id=3d5682e6868241d8904c217809d0ee52]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 08:49:01.449170 20517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6524b39b5724f048bd69d8d50745b99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:49:01.449214 20386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6524b39b5724f048bd69d8d50745b99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:49:01.449213 20647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6524b39b5724f048bd69d8d50745b99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:49:01.460968 20730 tablet_bootstrap.cc:492] T a6524b39b5724f048bd69d8d50745b99 P a753ff8fb0424e418de13498aac8da33: Bootstrap starting.
I20260928 08:49:01.464620 20730 tablet_bootstrap.cc:654] T a6524b39b5724f048bd69d8d50745b99 P a753ff8fb0424e418de13498aac8da33: Neither blocks nor log segments found. Creating new log.
I20260928 08:49:01.464594 20731 tablet_bootstrap.cc:492] T a6524b39b5724f048bd69d8d50745b99 P a7a5af83c69e4f91b3ff5294ead7d4f3: Bootstrap starting.
I20260928 08:49:01.466513 20730 log.cc:824] T a6524b39b5724f048bd69d8d50745b99 P a753ff8fb0424e418de13498aac8da33: Log is configured to *not* fsync() on all Append() calls
I20260928 08:49:01.468853 20731 tablet_bootstrap.cc:654] T a6524b39b5724f048bd69d8d50745b99 P a7a5af83c69e4f91b3ff5294ead7d4f3: Neither blocks nor log segments found. Creating new log.
I20260928 08:49:01.470664 20732 tablet_bootstrap.cc:492] T a6524b39b5724f048bd69d8d50745b99 P 399a25258b634c9696bb5c394ab07c46: Bootstrap starting.
I20260928 08:49:01.470651 20731 log.cc:824] T a6524b39b5724f048bd69d8d50745b99 P a7a5af83c69e4f91b3ff5294ead7d4f3: Log is configured to *not* fsync() on all Append() calls
I20260928 08:49:01.474191 20732 tablet_bootstrap.cc:654] T a6524b39b5724f048bd69d8d50745b99 P 399a25258b634c9696bb5c394ab07c46: Neither blocks nor log segments found. Creating new log.
I20260928 08:49:01.474375 20731 tablet_bootstrap.cc:492] T a6524b39b5724f048bd69d8d50745b99 P a7a5af83c69e4f91b3ff5294ead7d4f3: No bootstrap required, opened a new log
I20260928 08:49:01.474673 20731 ts_tablet_manager.cc:1397] T a6524b39b5724f048bd69d8d50745b99 P a7a5af83c69e4f91b3ff5294ead7d4f3: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 08:49:01.474872 20730 tablet_bootstrap.cc:492] T a6524b39b5724f048bd69d8d50745b99 P a753ff8fb0424e418de13498aac8da33: No bootstrap required, opened a new log
I20260928 08:49:01.475150 20730 ts_tablet_manager.cc:1397] T a6524b39b5724f048bd69d8d50745b99 P a753ff8fb0424e418de13498aac8da33: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.005s
I20260928 08:49:01.475386 20732 log.cc:824] T a6524b39b5724f048bd69d8d50745b99 P 399a25258b634c9696bb5c394ab07c46: Log is configured to *not* fsync() on all Append() calls
I20260928 08:49:01.477932 20732 tablet_bootstrap.cc:492] T a6524b39b5724f048bd69d8d50745b99 P 399a25258b634c9696bb5c394ab07c46: No bootstrap required, opened a new log
I20260928 08:49:01.478199 20732 ts_tablet_manager.cc:1397] T a6524b39b5724f048bd69d8d50745b99 P 399a25258b634c9696bb5c394ab07c46: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260928 08:49:01.486111 20732 raft_consensus.cc:348] T a6524b39b5724f048bd69d8d50745b99 P 399a25258b634c9696bb5c394ab07c46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a753ff8fb0424e418de13498aac8da33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46417 } } peers { permanent_uuid: "399a25258b634c9696bb5c394ab07c46" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44639 } } peers { permanent_uuid: "a7a5af83c69e4f91b3ff5294ead7d4f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42171 } }
I20260928 08:49:01.486142 20731 raft_consensus.cc:348] T a6524b39b5724f048bd69d8d50745b99 P a7a5af83c69e4f91b3ff5294ead7d4f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a753ff8fb0424e418de13498aac8da33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46417 } } peers { permanent_uuid: "399a25258b634c9696bb5c394ab07c46" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44639 } } peers { permanent_uuid: "a7a5af83c69e4f91b3ff5294ead7d4f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42171 } }
I20260928 08:49:01.486142 20730 raft_consensus.cc:348] T a6524b39b5724f048bd69d8d50745b99 P a753ff8fb0424e418de13498aac8da33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a753ff8fb0424e418de13498aac8da33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46417 } } peers { permanent_uuid: "399a25258b634c9696bb5c394ab07c46" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44639 } } peers { permanent_uuid: "a7a5af83c69e4f91b3ff5294ead7d4f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42171 } }
I20260928 08:49:01.486555 20732 raft_consensus.cc:374] T a6524b39b5724f048bd69d8d50745b99 P 399a25258b634c9696bb5c394ab07c46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:49:01.486613 20730 raft_consensus.cc:374] T a6524b39b5724f048bd69d8d50745b99 P a753ff8fb0424e418de13498aac8da33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:49:01.486635 20731 raft_consensus.cc:374] T a6524b39b5724f048bd69d8d50745b99 P a7a5af83c69e4f91b3ff5294ead7d4f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:49:01.486683 20732 raft_consensus.cc:729] T a6524b39b5724f048bd69d8d50745b99 P 399a25258b634c9696bb5c394ab07c46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 399a25258b634c9696bb5c394ab07c46, State: Initialized, Role: FOLLOWER
I20260928 08:49:01.486730 20731 raft_consensus.cc:729] T a6524b39b5724f048bd69d8d50745b99 P a7a5af83c69e4f91b3ff5294ead7d4f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7a5af83c69e4f91b3ff5294ead7d4f3, State: Initialized, Role: FOLLOWER
I20260928 08:49:01.486730 20730 raft_consensus.cc:729] T a6524b39b5724f048bd69d8d50745b99 P a753ff8fb0424e418de13498aac8da33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a753ff8fb0424e418de13498aac8da33, State: Initialized, Role: FOLLOWER
I20260928 08:49:01.487421 20731 consensus_queue.cc:260] T a6524b39b5724f048bd69d8d50745b99 P a7a5af83c69e4f91b3ff5294ead7d4f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a753ff8fb0424e418de13498aac8da33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46417 } } peers { permanent_uuid: "399a25258b634c9696bb5c394ab07c46" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44639 } } peers { permanent_uuid: "a7a5af83c69e4f91b3ff5294ead7d4f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42171 } }
I20260928 08:49:01.487452 20730 consensus_queue.cc:260] T a6524b39b5724f048bd69d8d50745b99 P a753ff8fb0424e418de13498aac8da33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a753ff8fb0424e418de13498aac8da33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46417 } } peers { permanent_uuid: "399a25258b634c9696bb5c394ab07c46" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44639 } } peers { permanent_uuid: "a7a5af83c69e4f91b3ff5294ead7d4f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42171 } }
I20260928 08:49:01.487478 20732 consensus_queue.cc:260] T a6524b39b5724f048bd69d8d50745b99 P 399a25258b634c9696bb5c394ab07c46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a753ff8fb0424e418de13498aac8da33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46417 } } peers { permanent_uuid: "399a25258b634c9696bb5c394ab07c46" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44639 } } peers { permanent_uuid: "a7a5af83c69e4f91b3ff5294ead7d4f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42171 } }
I20260928 08:49:01.488874 20582 heartbeater.cc:503] Master 127.0.97.62:40525 was elected leader, sending a full tablet report...
I20260928 08:49:01.489867 20730 ts_tablet_manager.cc:1428] T a6524b39b5724f048bd69d8d50745b99 P a753ff8fb0424e418de13498aac8da33: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260928 08:49:01.491715 20712 heartbeater.cc:503] Master 127.0.97.62:40525 was elected leader, sending a full tablet report...
I20260928 08:49:01.493325 20738 raft_consensus.cc:482] T a6524b39b5724f048bd69d8d50745b99 P a7a5af83c69e4f91b3ff5294ead7d4f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:49:01.493701 20738 raft_consensus.cc:504] T a6524b39b5724f048bd69d8d50745b99 P a7a5af83c69e4f91b3ff5294ead7d4f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a753ff8fb0424e418de13498aac8da33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46417 } } peers { permanent_uuid: "399a25258b634c9696bb5c394ab07c46" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44639 } } peers { permanent_uuid: "a7a5af83c69e4f91b3ff5294ead7d4f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42171 } }
I20260928 08:49:01.496258 20738 leader_election.cc:288] T a6524b39b5724f048bd69d8d50745b99 P a7a5af83c69e4f91b3ff5294ead7d4f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a753ff8fb0424e418de13498aac8da33 (127.0.97.2:46417), 399a25258b634c9696bb5c394ab07c46 (127.0.97.1:44639)
I20260928 08:49:01.497917 20737 raft_consensus.cc:482] T a6524b39b5724f048bd69d8d50745b99 P 399a25258b634c9696bb5c394ab07c46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 08:49:01.498024 20731 ts_tablet_manager.cc:1428] T a6524b39b5724f048bd69d8d50745b99 P a7a5af83c69e4f91b3ff5294ead7d4f3: Time spent starting tablet: real 0.023s user 0.018s sys 0.000s
I20260928 08:49:01.498289 20737 raft_consensus.cc:504] T a6524b39b5724f048bd69d8d50745b99 P 399a25258b634c9696bb5c394ab07c46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a753ff8fb0424e418de13498aac8da33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46417 } } peers { permanent_uuid: "399a25258b634c9696bb5c394ab07c46" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44639 } } peers { permanent_uuid: "a7a5af83c69e4f91b3ff5294ead7d4f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42171 } }
I20260928 08:49:01.498780 20732 ts_tablet_manager.cc:1428] T a6524b39b5724f048bd69d8d50745b99 P 399a25258b634c9696bb5c394ab07c46: Time spent starting tablet: real 0.020s user 0.014s sys 0.001s
I20260928 08:49:01.502420 20737 leader_election.cc:288] T a6524b39b5724f048bd69d8d50745b99 P 399a25258b634c9696bb5c394ab07c46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a753ff8fb0424e418de13498aac8da33 (127.0.97.2:46417), a7a5af83c69e4f91b3ff5294ead7d4f3 (127.0.97.3:42171)
I20260928 08:49:01.507956 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:01.499135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:59739 (local address 127.0.97.2:46417)
0928 08:49:01.499364 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:01.499370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:01.499653 (+ 283us) server_negotiation.cc:407] Connection header received
0928 08:49:01.499728 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:01.499733 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:01.499881 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:01.499968 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:49:01.500609 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.501626 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.504588 (+ 2962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.506225 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.506394 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:01.506530 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:01.507360 (+ 830us) server_negotiation.cc:299] Negotiation successful
0928 08:49:01.507547 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 08:49:01.507982 20714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:01.499029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46417 (local address 127.0.97.3:59739)
0928 08:49:01.499350 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:01.499372 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:49:01.499526 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:49:01.500164 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:49:01.500169 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:49:01.500195 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:49:01.500401 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:01.500415 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.501968 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.501974 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:01.504313 (+ 2339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:01.504336 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.506680 (+ 2344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.506689 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:01.506991 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:01.507046 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 08:49:01.507158 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 08:49:01.507512 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":99}
I20260928 08:49:01.509586 20537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6524b39b5724f048bd69d8d50745b99" candidate_uuid: "a7a5af83c69e4f91b3ff5294ead7d4f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a753ff8fb0424e418de13498aac8da33" is_pre_election: true
I20260928 08:49:01.510331 20537 raft_consensus.cc:2454] T a6524b39b5724f048bd69d8d50745b99 P a753ff8fb0424e418de13498aac8da33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7a5af83c69e4f91b3ff5294ead7d4f3 in term 0.
I20260928 08:49:01.511797 20603 leader_election.cc:302] T a6524b39b5724f048bd69d8d50745b99 P a7a5af83c69e4f91b3ff5294ead7d4f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a753ff8fb0424e418de13498aac8da33, a7a5af83c69e4f91b3ff5294ead7d4f3; no voters:
I20260928 08:49:01.512494 20739 raft_consensus.cc:2781] T a6524b39b5724f048bd69d8d50745b99 P a7a5af83c69e4f91b3ff5294ead7d4f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 08:49:01.512660 20739 raft_consensus.cc:482] T a6524b39b5724f048bd69d8d50745b99 P a7a5af83c69e4f91b3ff5294ead7d4f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 08:49:01.512806 20739 raft_consensus.cc:3037] T a6524b39b5724f048bd69d8d50745b99 P a7a5af83c69e4f91b3ff5294ead7d4f3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:49:01.514714 20728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:01.499513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.3:46865 (local address 127.0.97.1:44639)
0928 08:49:01.499734 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:01.499743 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:01.503333 (+ 3590us) server_negotiation.cc:407] Connection header received
0928 08:49:01.503549 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:01.503555 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:01.503683 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:01.503776 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:49:01.504906 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.506374 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.511032 (+ 4658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.512939 (+ 1907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.513100 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:01.513257 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:01.514095 (+ 838us) server_negotiation.cc:299] Negotiation successful
0928 08:49:01.514315 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 08:49:01.515774 20740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:01.499503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44639 (local address 127.0.97.3:46865)
0928 08:49:01.503217 (+ 3714us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:01.503233 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 08:49:01.503404 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:49:01.504053 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:49:01.504060 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:49:01.504113 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:49:01.504667 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:01.504687 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.506702 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.506710 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:01.510675 (+ 3965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:01.510705 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.513495 (+ 2790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.513505 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:01.513799 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:01.513857 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 08:49:01.514944 (+ 1087us) client_negotiation.cc:240] Negotiation successful
0928 08:49:01.515193 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3578,"mutex_wait_us":1,"thread_start_us":3384,"threads_started":1}
I20260928 08:49:01.518771 20739 raft_consensus.cc:504] T a6524b39b5724f048bd69d8d50745b99 P a7a5af83c69e4f91b3ff5294ead7d4f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a753ff8fb0424e418de13498aac8da33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46417 } } peers { permanent_uuid: "399a25258b634c9696bb5c394ab07c46" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44639 } } peers { permanent_uuid: "a7a5af83c69e4f91b3ff5294ead7d4f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42171 } }
I20260928 08:49:01.519989 20739 leader_election.cc:288] T a6524b39b5724f048bd69d8d50745b99 P a7a5af83c69e4f91b3ff5294ead7d4f3 [CANDIDATE]: Term 1 election: Requested vote from peers a753ff8fb0424e418de13498aac8da33 (127.0.97.2:46417), 399a25258b634c9696bb5c394ab07c46 (127.0.97.1:44639)
I20260928 08:49:01.520531 20537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6524b39b5724f048bd69d8d50745b99" candidate_uuid: "a7a5af83c69e4f91b3ff5294ead7d4f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a753ff8fb0424e418de13498aac8da33"
I20260928 08:49:01.520784 20537 raft_consensus.cc:3037] T a6524b39b5724f048bd69d8d50745b99 P a753ff8fb0424e418de13498aac8da33 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:49:01.520957 20405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6524b39b5724f048bd69d8d50745b99" candidate_uuid: "a7a5af83c69e4f91b3ff5294ead7d4f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "399a25258b634c9696bb5c394ab07c46"
I20260928 08:49:01.521456 20405 raft_consensus.cc:3037] T a6524b39b5724f048bd69d8d50745b99 P 399a25258b634c9696bb5c394ab07c46 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:49:01.524458 20537 raft_consensus.cc:2454] T a6524b39b5724f048bd69d8d50745b99 P a753ff8fb0424e418de13498aac8da33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7a5af83c69e4f91b3ff5294ead7d4f3 in term 1.
I20260928 08:49:01.525007 20603 leader_election.cc:302] T a6524b39b5724f048bd69d8d50745b99 P a7a5af83c69e4f91b3ff5294ead7d4f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a753ff8fb0424e418de13498aac8da33, a7a5af83c69e4f91b3ff5294ead7d4f3; no voters:
I20260928 08:49:01.525324 20739 raft_consensus.cc:2781] T a6524b39b5724f048bd69d8d50745b99 P a7a5af83c69e4f91b3ff5294ead7d4f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:49:01.525205 20406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6524b39b5724f048bd69d8d50745b99" candidate_uuid: "a7a5af83c69e4f91b3ff5294ead7d4f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "399a25258b634c9696bb5c394ab07c46" is_pre_election: true
I20260928 08:49:01.525698 20739 raft_consensus.cc:686] T a6524b39b5724f048bd69d8d50745b99 P a7a5af83c69e4f91b3ff5294ead7d4f3 [term 1 LEADER]: Becoming Leader. State: Replica: a7a5af83c69e4f91b3ff5294ead7d4f3, State: Running, Role: LEADER
I20260928 08:49:01.525728 20405 raft_consensus.cc:2454] T a6524b39b5724f048bd69d8d50745b99 P 399a25258b634c9696bb5c394ab07c46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7a5af83c69e4f91b3ff5294ead7d4f3 in term 1.
I20260928 08:49:01.526296 20739 consensus_queue.cc:237] T a6524b39b5724f048bd69d8d50745b99 P a7a5af83c69e4f91b3ff5294ead7d4f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a753ff8fb0424e418de13498aac8da33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46417 } } peers { permanent_uuid: "399a25258b634c9696bb5c394ab07c46" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44639 } } peers { permanent_uuid: "a7a5af83c69e4f91b3ff5294ead7d4f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42171 } }
I20260928 08:49:01.530256 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:01.503275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:36443 (local address 127.0.97.3:42171)
0928 08:49:01.503448 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:01.503453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:01.520726 (+ 17273us) server_negotiation.cc:407] Connection header received
0928 08:49:01.521055 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:01.521062 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:01.521144 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:01.521217 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:49:01.522063 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.523178 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.527947 (+ 4769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.529231 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.529385 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:01.529496 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:01.529823 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 08:49:01.529979 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 08:49:01.530402 20743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:01.502730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.3:42171 (local address 127.0.97.1:36443)
0928 08:49:01.520571 (+ 17841us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:01.520591 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:49:01.520850 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:49:01.521412 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:49:01.521418 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:49:01.521454 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:49:01.521869 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:01.521899 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.523367 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.523376 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:01.527704 (+ 4328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:01.527720 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.529444 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.529450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:01.529621 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:01.529666 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 08:49:01.529778 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 08:49:01.529957 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17673,"mutex_wait_us":145,"thread_start_us":17092,"threads_started":1}
I20260928 08:49:01.530783 20741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:01.504190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.2:46417 (local address 127.0.97.1:34093)
0928 08:49:01.521147 (+ 16957us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:01.521164 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 08:49:01.521341 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:49:01.521976 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:49:01.521980 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:49:01.522006 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 08:49:01.522434 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:01.522511 (+ 77us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.524139 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.524145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:01.528145 (+ 4000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:01.528162 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.529959 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.529965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:01.530134 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:01.530165 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 08:49:01.530265 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 08:49:01.530419 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16827,"mutex_wait_us":72,"thread_start_us":15194,"threads_started":1}
I20260928 08:49:01.531419 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:01.504122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.97.1:34093 (local address 127.0.97.2:46417)
0928 08:49:01.511409 (+ 7287us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:01.511418 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:01.521243 (+ 9825us) server_negotiation.cc:407] Connection header received
0928 08:49:01.521604 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:01.521610 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:01.521686 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:01.521756 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 08:49:01.522710 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.523947 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.528393 (+ 4446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.529584 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.529672 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:01.529772 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:01.530989 (+ 1217us) server_negotiation.cc:299] Negotiation successful
0928 08:49:01.531105 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7127,"thread_start_us":15543,"threads_started":1}
I20260928 08:49:01.532009 20667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6524b39b5724f048bd69d8d50745b99" candidate_uuid: "399a25258b634c9696bb5c394ab07c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7a5af83c69e4f91b3ff5294ead7d4f3" is_pre_election: true
I20260928 08:49:01.532544 20264 catalog_manager.cc:5495] T a6524b39b5724f048bd69d8d50745b99 P a7a5af83c69e4f91b3ff5294ead7d4f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a7a5af83c69e4f91b3ff5294ead7d4f3 (127.0.97.3). New cstate: current_term: 1 leader_uuid: "a7a5af83c69e4f91b3ff5294ead7d4f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a753ff8fb0424e418de13498aac8da33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "399a25258b634c9696bb5c394ab07c46" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7a5af83c69e4f91b3ff5294ead7d4f3" member_type: VOTER last_known_addr { host: "127.0.97.3" port: 42171 } health_report { overall_health: HEALTHY } } }
I20260928 08:49:01.534796 20537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6524b39b5724f048bd69d8d50745b99" candidate_uuid: "399a25258b634c9696bb5c394ab07c46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a753ff8fb0424e418de13498aac8da33" is_pre_election: true
I20260928 08:49:01.535147 20537 raft_consensus.cc:2379] T a6524b39b5724f048bd69d8d50745b99 P a753ff8fb0424e418de13498aac8da33 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 399a25258b634c9696bb5c394ab07c46 in current term 1: Already voted for candidate a7a5af83c69e4f91b3ff5294ead7d4f3 in this term.
I20260928 08:49:01.535964 20342 leader_election.cc:302] T a6524b39b5724f048bd69d8d50745b99 P 399a25258b634c9696bb5c394ab07c46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 399a25258b634c9696bb5c394ab07c46; no voters: a753ff8fb0424e418de13498aac8da33, a7a5af83c69e4f91b3ff5294ead7d4f3
I20260928 08:49:01.536388 20737 raft_consensus.cc:2726] T a6524b39b5724f048bd69d8d50745b99 P 399a25258b634c9696bb5c394ab07c46 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 08:49:01.555601 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:01.550097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46700 (local address 127.0.97.3:42171)
0928 08:49:01.550271 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:01.550275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:01.550295 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:49:01.550479 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:01.550483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:01.550582 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:01.550709 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:49:01.551034 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.551791 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.553256 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.553713 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.553811 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:01.554380 (+ 569us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:49:01.554967 (+ 587us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:49:01.555083 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:01.555171 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 08:49:01.555242 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 08:49:01.564230 20405 raft_consensus.cc:1261] T a6524b39b5724f048bd69d8d50745b99 P 399a25258b634c9696bb5c394ab07c46 [term 1 FOLLOWER]: Refusing update from remote peer a7a5af83c69e4f91b3ff5294ead7d4f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:49:01.564270 20537 raft_consensus.cc:1261] T a6524b39b5724f048bd69d8d50745b99 P a753ff8fb0424e418de13498aac8da33 [term 1 FOLLOWER]: Refusing update from remote peer a7a5af83c69e4f91b3ff5294ead7d4f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:49:01.565165 20739 consensus_queue.cc:1035] T a6524b39b5724f048bd69d8d50745b99 P a7a5af83c69e4f91b3ff5294ead7d4f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "399a25258b634c9696bb5c394ab07c46" member_type: VOTER last_known_addr { host: "127.0.97.1" port: 44639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:49:01.565480 20738 consensus_queue.cc:1035] T a6524b39b5724f048bd69d8d50745b99 P a7a5af83c69e4f91b3ff5294ead7d4f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a753ff8fb0424e418de13498aac8da33" member_type: VOTER last_known_addr { host: "127.0.97.2" port: 46417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:49:01.581483 20747 mvcc.cc:205] Tried to move back new op lower bound from 7334237559036760064 to 7334237558897266688. Current Snapshot: MvccSnapshot[applied={T|T < 7334237559036760064}]
I20260928 08:49:01.600921 20264 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:49:01.601221 20264 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41090:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 08:49:01.605592 20264 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ae0259e51214be486f8af5608b73a79: Sending DeleteTablet for 3 replicas of tablet a6524b39b5724f048bd69d8d50745b99
I20260928 08:49:01.607012 20647 tablet_service.cc:1514] Processing DeleteTablet for tablet a6524b39b5724f048bd69d8d50745b99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:49:01 UTC) from {username='slave'} at 127.0.0.1:46688
I20260928 08:49:01.607343 20386 tablet_service.cc:1514] Processing DeleteTablet for tablet a6524b39b5724f048bd69d8d50745b99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:49:01 UTC) from {username='slave'} at 127.0.0.1:54246
I20260928 08:49:01.607329 20517 tablet_service.cc:1514] Processing DeleteTablet for tablet a6524b39b5724f048bd69d8d50745b99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 08:49:01 UTC) from {username='slave'} at 127.0.0.1:57014
I20260928 08:49:01.608165 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 20324
I20260928 08:49:01.628327 20756 tablet_replica.cc:330] T a6524b39b5724f048bd69d8d50745b99 P a7a5af83c69e4f91b3ff5294ead7d4f3: stopping tablet replica
I20260928 08:49:01.628963 20757 tablet_replica.cc:330] T a6524b39b5724f048bd69d8d50745b99 P 399a25258b634c9696bb5c394ab07c46: stopping tablet replica
I20260928 08:49:01.629405 20757 raft_consensus.cc:2229] T a6524b39b5724f048bd69d8d50745b99 P 399a25258b634c9696bb5c394ab07c46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:49:01.629905 20757 raft_consensus.cc:2258] T a6524b39b5724f048bd69d8d50745b99 P 399a25258b634c9696bb5c394ab07c46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:49:01.632087 20756 raft_consensus.cc:2229] T a6524b39b5724f048bd69d8d50745b99 P a7a5af83c69e4f91b3ff5294ead7d4f3 [term 1 LEADER]: Raft consensus shutting down.
I20260928 08:49:01.632802 20756 raft_consensus.cc:2258] T a6524b39b5724f048bd69d8d50745b99 P a7a5af83c69e4f91b3ff5294ead7d4f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:49:01.635490 20728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:01.610222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54250 (local address 127.0.97.1:44639)
0928 08:49:01.610472 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:01.610482 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:01.621659 (+ 11177us) server_negotiation.cc:407] Connection header received
0928 08:49:01.622142 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:01.622153 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:01.622349 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:01.622451 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:01.623454 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.625080 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.630334 (+ 5254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.630793 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.630885 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:01.633575 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:01.633588 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:01.633606 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:49:01.634458 (+ 852us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:01.634529 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:49:01.634559 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:01.634701 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:01.634957 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 08:49:01.635109 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 08:49:01.637012 20446 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:49:01.638175 20756 ts_tablet_manager.cc:1905] T a6524b39b5724f048bd69d8d50745b99 P a7a5af83c69e4f91b3ff5294ead7d4f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:49:01.639016 20758 tablet_replica.cc:330] T a6524b39b5724f048bd69d8d50745b99 P a753ff8fb0424e418de13498aac8da33: stopping tablet replica
I20260928 08:49:01.639741 20758 raft_consensus.cc:2229] T a6524b39b5724f048bd69d8d50745b99 P a753ff8fb0424e418de13498aac8da33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 08:49:01.640980 20758 raft_consensus.cc:2258] T a6524b39b5724f048bd69d8d50745b99 P a753ff8fb0424e418de13498aac8da33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 08:49:01.644170 20758 ts_tablet_manager.cc:1905] T a6524b39b5724f048bd69d8d50745b99 P a753ff8fb0424e418de13498aac8da33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:49:01.647532 20756 ts_tablet_manager.cc:1918] T a6524b39b5724f048bd69d8d50745b99 P a7a5af83c69e4f91b3ff5294ead7d4f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:49:01.647909 20756 log.cc:1196] T a6524b39b5724f048bd69d8d50745b99 P a7a5af83c69e4f91b3ff5294ead7d4f3: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-2/wal/wals/a6524b39b5724f048bd69d8d50745b99
I20260928 08:49:01.648602 20756 ts_tablet_manager.cc:1939] T a6524b39b5724f048bd69d8d50745b99 P a7a5af83c69e4f91b3ff5294ead7d4f3: Deleting consensus metadata
I20260928 08:49:01.650832 20249 catalog_manager.cc:4841] TS a7a5af83c69e4f91b3ff5294ead7d4f3 (127.0.97.3:42171): tablet a6524b39b5724f048bd69d8d50745b99 (table test-table [id=3d5682e6868241d8904c217809d0ee52]) successfully deleted
I20260928 08:49:01.655164 20758 ts_tablet_manager.cc:1918] T a6524b39b5724f048bd69d8d50745b99 P a753ff8fb0424e418de13498aac8da33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 08:49:01.655393 20758 log.cc:1196] T a6524b39b5724f048bd69d8d50745b99 P a753ff8fb0424e418de13498aac8da33: Deleting WAL directory at /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790585022442112-388-0/minicluster-data/ts-1/wal/wals/a6524b39b5724f048bd69d8d50745b99
I20260928 08:49:01.656100 20758 ts_tablet_manager.cc:1939] T a6524b39b5724f048bd69d8d50745b99 P a753ff8fb0424e418de13498aac8da33: Deleting consensus metadata
I20260928 08:49:01.658365 20252 catalog_manager.cc:4841] TS a753ff8fb0424e418de13498aac8da33 (127.0.97.2:46417): tablet a6524b39b5724f048bd69d8d50745b99 (table test-table [id=3d5682e6868241d8904c217809d0ee52]) successfully deleted
I20260928 08:49:01.638813 20757 ts_tablet_manager.cc:1905] T a6524b39b5724f048bd69d8d50745b99 P 399a25258b634c9696bb5c394ab07c46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 08:49:01.744633 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 20324
W20260928 08:49:01.763882 20252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.97.1:44639 (error 108)
I20260928 08:49:01.764328 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 20455
I20260928 08:49:01.776911 20729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:01.765628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57030 (local address 127.0.97.2:46417)
0928 08:49:01.765823 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:01.765830 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:01.765853 (+ 23us) server_negotiation.cc:407] Connection header received
0928 08:49:01.766095 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:01.766104 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:01.766255 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:01.766370 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:01.767211 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.768719 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.771229 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.772158 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.772330 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:01.774280 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:01.774297 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:01.774325 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:49:01.775363 (+ 1038us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:01.775457 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:49:01.775491 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:01.775634 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:01.776439 (+ 805us) server_negotiation.cc:299] Negotiation successful
0928 08:49:01.776561 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 08:49:01.777801 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:01.777248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44639 (local address 127.0.0.1:54262)
0928 08:49:01.777454 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:01.777606 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 08:49:01.777882 20577 generic_service.cc:196] Checking for leaks (request via RPC)
W20260928 08:49:01.783105 20252 catalog_manager.cc:4584] TS 399a25258b634c9696bb5c394ab07c46 (127.0.97.1:44639): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a6524b39b5724f048bd69d8d50745b99: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44639: connect: Connection refused (error 111)
I20260928 08:49:01.837421 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:01.836792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44639 (local address 127.0.0.1:54266)
0928 08:49:01.837076 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:01.837201 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260928 08:49:01.838699 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:01.838250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44639 (local address 127.0.0.1:54278)
0928 08:49:01.838435 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:01.838541 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 08:49:01.877158 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 20455
I20260928 08:49:01.895205 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 20585
I20260928 08:49:01.900286 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:01.899549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44639 (local address 127.0.0.1:54294)
0928 08:49:01.899907 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:01.900025 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260928 08:49:01.901911 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:01.901348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44639 (local address 127.0.0.1:54296)
0928 08:49:01.901608 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:01.901705 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260928 08:49:01.905375 20727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:01.896122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46716 (local address 127.0.97.3:42171)
0928 08:49:01.896311 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:01.896315 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:01.896440 (+ 125us) server_negotiation.cc:407] Connection header received
0928 08:49:01.896626 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:01.896630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:01.896750 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:01.896840 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:01.897415 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.898270 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.900453 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:01.901150 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:01.901325 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:01.903024 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:01.903044 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:01.903068 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:49:01.904200 (+ 1132us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:01.904305 (+ 105us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:49:01.904415 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:01.904556 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:01.904820 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 08:49:01.904993 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 08:49:01.906214 20707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:49:01.982926 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:01.982328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44639 (local address 127.0.0.1:54310)
0928 08:49:01.982612 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:01.982732 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260928 08:49:01.984186 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:01.983718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44639 (local address 127.0.0.1:54320)
0928 08:49:01.983962 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:01.984039 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260928 08:49:02.032989 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 20585
I20260928 08:49:02.055034 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 20234
I20260928 08:49:02.056332 20295 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:49:02.170459 20725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:02.169693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.1:44639 (local address 127.0.0.1:54334)
0928 08:49:02.169991 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:02.170162 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.1:44639: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 08:49:02.170644 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 20234
2026-09-28T08:49:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2943 ms)
[----------] 6 tests from EncryptionPolicyTest (23454 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:49:02 dist-test-slave-n6xs krb5kdc[20767](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:49:02 dist-test-slave-n6xs krb5kdc[20767](info): set up 2 sockets
Sep 28 08:49:02 dist-test-slave-n6xs krb5kdc[20767](info): commencing operation
krb5kdc: starting...
W20260928 08:49:03.292279 388 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:49:03 dist-test-slave-n6xs krb5kdc[20767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:49:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:49:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:49:03.567570 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35887
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41707
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:49:03.916785 20780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:49:03.917099 20780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:49:03.917304 20780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:49:03.924913 20780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:49:03.925072 20780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:49:03.925156 20780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:49:03.925237 20780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:49:03.925314 20780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:49:03.935724 20780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41707
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35887
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:49:03.937178 20780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:49:03.938536 20780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:49:03.951393 20786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:03.951457 20788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:03.951442 20785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:49:03.952749 20780 server_base.cc:1034] running on GCE node
I20260928 08:49:03.953632 20780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:49:03.955241 20780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:49:03.956491 20780 hybrid_clock.cc:648] HybridClock initialized: now 1790585343956454 us; error 58 us; skew 500 ppm
I20260928 08:49:03.956880 20780 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:49:03 dist-test-slave-n6xs krb5kdc[20767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585343, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:49:03.960245 20780 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:49:03.961961 20780 webserver.cc:460] Webserver started at http://127.0.0.1:42927/ using document root <none> and password file <none>
I20260928 08:49:03.962702 20780 fs_manager.cc:359] Metadata directory not provided
I20260928 08:49:03.962857 20780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:49:03.963177 20780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:49:03.966487 20780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "509ffe3c22b64c439c9c6423e4fc135c"
format_stamp: "Formatted at 2026-09-28 08:49:03 on dist-test-slave-n6xs"
server_key: "457c3ca056f47200e4405a83f1eaa009"
server_key_iv: "e4632c0d6b34d97e676410c16394dff3"
server_key_version: "encryptionkey@0"
I20260928 08:49:03.967339 20780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "509ffe3c22b64c439c9c6423e4fc135c"
format_stamp: "Formatted at 2026-09-28 08:49:03 on dist-test-slave-n6xs"
server_key: "457c3ca056f47200e4405a83f1eaa009"
server_key_iv: "e4632c0d6b34d97e676410c16394dff3"
server_key_version: "encryptionkey@0"
I20260928 08:49:03.973296 20780 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 08:49:03.977475 20795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:49:03.979113 20780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:49:03.979359 20780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "509ffe3c22b64c439c9c6423e4fc135c"
format_stamp: "Formatted at 2026-09-28 08:49:03 on dist-test-slave-n6xs"
server_key: "457c3ca056f47200e4405a83f1eaa009"
server_key_iv: "e4632c0d6b34d97e676410c16394dff3"
server_key_version: "encryptionkey@0"
I20260928 08:49:03.979602 20780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:49:03.997660 20780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:49:04.002251 20780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:49:04.002570 20780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:49:04.022315 20780 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35887
I20260928 08:49:04.022336 20846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35887 every 8 connection(s)
I20260928 08:49:04.023916 20780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:49:04.024631 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 20780
I20260928 08:49:04.024945 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:49:04.025735 388 external_mini_cluster.cc:1428] Setting key 6f56168a7cde582ace6a70a9dbc08a23
I20260928 08:49:04.028867 20847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 08:49:04 dist-test-slave-n6xs krb5kdc[20767](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 08:49:04.042909 20847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 509ffe3c22b64c439c9c6423e4fc135c: Bootstrap starting.
I20260928 08:49:04.047480 20847 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 509ffe3c22b64c439c9c6423e4fc135c: Neither blocks nor log segments found. Creating new log.
I20260928 08:49:04.048904 20847 log.cc:824] T 00000000000000000000000000000000 P 509ffe3c22b64c439c9c6423e4fc135c: Log is configured to *not* fsync() on all Append() calls
I20260928 08:49:04.053022 20847 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 509ffe3c22b64c439c9c6423e4fc135c: No bootstrap required, opened a new log
I20260928 08:49:04.054003 20850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:04.027915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48241 (local address 127.0.0.1:35887)
0928 08:49:04.028765 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:04.028783 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:04.028854 (+ 71us) server_negotiation.cc:407] Connection header received
0928 08:49:04.031249 (+ 2395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:04.031276 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:04.031792 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:04.032357 (+ 565us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:04.033828 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:04.035982 (+ 2154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:04.037406 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:04.038790 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:04.039010 (+ 220us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:04.043407 (+ 4397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:04.043420 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:04.043441 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:04.043493 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:04.047756 (+ 4263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:04.048723 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:04.048732 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:04.048741 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:04.049011 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:04.049630 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:04.049638 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:04.049643 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:04.050597 (+ 954us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:04.051704 (+ 1107us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:04.052093 (+ 389us) server_negotiation.cc:299] Negotiation successful
0928 08:49:04.052448 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":385,"threads_started":1}
I20260928 08:49:04.058563 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:49:04.061897 20847 raft_consensus.cc:348] T 00000000000000000000000000000000 P 509ffe3c22b64c439c9c6423e4fc135c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "509ffe3c22b64c439c9c6423e4fc135c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35887 } }
I20260928 08:49:04.062342 20847 raft_consensus.cc:374] T 00000000000000000000000000000000 P 509ffe3c22b64c439c9c6423e4fc135c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:49:04.062453 20847 raft_consensus.cc:729] T 00000000000000000000000000000000 P 509ffe3c22b64c439c9c6423e4fc135c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 509ffe3c22b64c439c9c6423e4fc135c, State: Initialized, Role: FOLLOWER
I20260928 08:49:04.063038 20847 consensus_queue.cc:260] T 00000000000000000000000000000000 P 509ffe3c22b64c439c9c6423e4fc135c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "509ffe3c22b64c439c9c6423e4fc135c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35887 } }
I20260928 08:49:04.063319 20847 raft_consensus.cc:388] T 00000000000000000000000000000000 P 509ffe3c22b64c439c9c6423e4fc135c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:49:04.063512 20847 raft_consensus.cc:482] T 00000000000000000000000000000000 P 509ffe3c22b64c439c9c6423e4fc135c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:49:04.063707 20847 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 509ffe3c22b64c439c9c6423e4fc135c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:49:04.066354 20847 raft_consensus.cc:504] T 00000000000000000000000000000000 P 509ffe3c22b64c439c9c6423e4fc135c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "509ffe3c22b64c439c9c6423e4fc135c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35887 } }
I20260928 08:49:04.067020 20847 leader_election.cc:302] T 00000000000000000000000000000000 P 509ffe3c22b64c439c9c6423e4fc135c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 509ffe3c22b64c439c9c6423e4fc135c; no voters:
I20260928 08:49:04.067768 20847 leader_election.cc:288] T 00000000000000000000000000000000 P 509ffe3c22b64c439c9c6423e4fc135c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:49:04.067999 20857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 509ffe3c22b64c439c9c6423e4fc135c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:49:04.069202 20857 raft_consensus.cc:686] T 00000000000000000000000000000000 P 509ffe3c22b64c439c9c6423e4fc135c [term 1 LEADER]: Becoming Leader. State: Replica: 509ffe3c22b64c439c9c6423e4fc135c, State: Running, Role: LEADER
I20260928 08:49:04.070003 20857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 509ffe3c22b64c439c9c6423e4fc135c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "509ffe3c22b64c439c9c6423e4fc135c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35887 } }
I20260928 08:49:04.070182 20847 sys_catalog.cc:564] T 00000000000000000000000000000000 P 509ffe3c22b64c439c9c6423e4fc135c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:49:04.073035 20859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 509ffe3c22b64c439c9c6423e4fc135c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 509ffe3c22b64c439c9c6423e4fc135c. Latest consensus state: current_term: 1 leader_uuid: "509ffe3c22b64c439c9c6423e4fc135c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "509ffe3c22b64c439c9c6423e4fc135c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35887 } } }
I20260928 08:49:04.073460 20859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 509ffe3c22b64c439c9c6423e4fc135c [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:04.074095 20858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 509ffe3c22b64c439c9c6423e4fc135c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "509ffe3c22b64c439c9c6423e4fc135c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "509ffe3c22b64c439c9c6423e4fc135c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35887 } } }
I20260928 08:49:04.074478 20858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 509ffe3c22b64c439c9c6423e4fc135c [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:04.085767 20850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:04.063610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37737 (local address 127.0.0.1:35887)
0928 08:49:04.063843 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:04.063852 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:04.064309 (+ 457us) server_negotiation.cc:407] Connection header received
0928 08:49:04.064459 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:04.064465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:04.064545 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:04.064673 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:04.066373 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:04.067529 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:04.069784 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:04.070514 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:04.070649 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:04.074696 (+ 4047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:04.074706 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:04.074728 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:04.074788 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:04.079663 (+ 4875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:04.081274 (+ 1611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:04.081282 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:04.081289 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:04.081538 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:04.082301 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:04.082309 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:04.082315 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:04.082805 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:04.084617 (+ 1812us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:04.084863 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 08:49:04.085120 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 08:49:04.097214 20863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:49:04.097432 20874 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 08:49:04.102124 20863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:49:04.110749 20863 catalog_manager.cc:1347] Generated new cluster ID: 42ae6514b44b4aa1bf7c8b8c532f1602
I20260928 08:49:04.110888 20863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:49:04.120253 20863 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:49:04.122198 20863 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:49:04.136883 20863 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 509ffe3c22b64c439c9c6423e4fc135c: Generated new TSK 0
I20260928 08:49:04.138124 20863 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:49:04.153074 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 20780
I20260928 08:49:04.154347 20842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:49:04.248292 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 20780
W20260928 08:49:04.266172 20765 connection.cc:535] client connection to 127.0.0.1:35887 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35887): Connection reset by peer (error 104)
2026-09-28T08:49:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2076 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:49:04 dist-test-slave-n6xs krb5kdc[20880](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:49:04 dist-test-slave-n6xs krb5kdc[20880](info): set up 2 sockets
Sep 28 08:49:04 dist-test-slave-n6xs krb5kdc[20880](info): commencing operation
krb5kdc: starting...
W20260928 08:49:05.364177 388 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:49:05 dist-test-slave-n6xs krb5kdc[20880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:49:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:49:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:49:05.635505 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44647
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:49:05.978871 20893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:49:05.979204 20893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:49:05.979296 20893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:49:05.986894 20893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:49:05.987033 20893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:49:05.987082 20893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:49:05.987143 20893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:49:05.987190 20893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:49:05.997404 20893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44647
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:49:05.998837 20893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:49:06.000178 20893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:49:06.012595 20898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:06.012598 20901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:06.012657 20899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:49:06.014407 20893 server_base.cc:1034] running on GCE node
I20260928 08:49:06.015338 20893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:49:06.016956 20893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:49:06.018198 20893 hybrid_clock.cc:648] HybridClock initialized: now 1790585346018148 us; error 45 us; skew 500 ppm
I20260928 08:49:06.018569 20893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:49:06 dist-test-slave-n6xs krb5kdc[20880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:49:06.028388 20893 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:49:06.030121 20893 webserver.cc:460] Webserver started at http://127.0.0.1:43239/ using document root <none> and password file <none>
I20260928 08:49:06.030858 20893 fs_manager.cc:359] Metadata directory not provided
I20260928 08:49:06.031005 20893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:49:06.031323 20893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:49:06.034498 20893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "a36d6431a73943f894c8e9b8ae4da145"
format_stamp: "Formatted at 2026-09-28 08:49:06 on dist-test-slave-n6xs"
server_key: "cb148aa3d97a27c18c326f8deef1e72d"
server_key_iv: "2493d9584b95104359a543ae780a37e9"
server_key_version: "encryptionkey@0"
I20260928 08:49:06.035274 20893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "a36d6431a73943f894c8e9b8ae4da145"
format_stamp: "Formatted at 2026-09-28 08:49:06 on dist-test-slave-n6xs"
server_key: "cb148aa3d97a27c18c326f8deef1e72d"
server_key_iv: "2493d9584b95104359a543ae780a37e9"
server_key_version: "encryptionkey@0"
I20260928 08:49:06.040867 20893 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260928 08:49:06.044514 20908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:49:06.045895 20893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:49:06.046113 20893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "a36d6431a73943f894c8e9b8ae4da145"
format_stamp: "Formatted at 2026-09-28 08:49:06 on dist-test-slave-n6xs"
server_key: "cb148aa3d97a27c18c326f8deef1e72d"
server_key_iv: "2493d9584b95104359a543ae780a37e9"
server_key_version: "encryptionkey@0"
I20260928 08:49:06.046348 20893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:49:06.068486 20893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:49:06.073143 20893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:49:06.073450 20893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:49:06.092768 20893 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44647
I20260928 08:49:06.092818 20959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44647 every 8 connection(s)
I20260928 08:49:06.094262 20893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:49:06.098269 20960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:49:06.102489 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 20893
I20260928 08:49:06.102696 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:49:06.103153 388 external_mini_cluster.cc:1428] Setting key e13ea089f3500deba61845a7c4dbcd07
I20260928 08:49:06.109692 20960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a36d6431a73943f894c8e9b8ae4da145: Bootstrap starting.
I20260928 08:49:06.113579 20960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a36d6431a73943f894c8e9b8ae4da145: Neither blocks nor log segments found. Creating new log.
I20260928 08:49:06.115013 20960 log.cc:824] T 00000000000000000000000000000000 P a36d6431a73943f894c8e9b8ae4da145: Log is configured to *not* fsync() on all Append() calls
I20260928 08:49:06.118296 20963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:06.104950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46597 (local address 127.0.0.1:44647)
0928 08:49:06.105759 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:06.105771 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:06.105811 (+ 40us) server_negotiation.cc:407] Connection header received
0928 08:49:06.108110 (+ 2299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:06.108135 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:06.108571 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:06.108937 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:06.109582 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:06.110954 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:06.113034 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:06.113853 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:06.114022 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:06.115434 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:06.115443 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:06.115460 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:49:06.116395 (+ 935us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:06.116508 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:49:06.116549 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:06.116717 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:06.117114 (+ 397us) server_negotiation.cc:299] Negotiation successful
0928 08:49:06.117328 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":377,"threads_started":1}
I20260928 08:49:06.118525 20960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a36d6431a73943f894c8e9b8ae4da145: No bootstrap required, opened a new log
I20260928 08:49:06.122400 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:49:06.126935 20960 raft_consensus.cc:348] T 00000000000000000000000000000000 P a36d6431a73943f894c8e9b8ae4da145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a36d6431a73943f894c8e9b8ae4da145" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44647 } }
I20260928 08:49:06.127455 20960 raft_consensus.cc:374] T 00000000000000000000000000000000 P a36d6431a73943f894c8e9b8ae4da145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:49:06.127585 20960 raft_consensus.cc:729] T 00000000000000000000000000000000 P a36d6431a73943f894c8e9b8ae4da145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a36d6431a73943f894c8e9b8ae4da145, State: Initialized, Role: FOLLOWER
I20260928 08:49:06.128299 20960 consensus_queue.cc:260] T 00000000000000000000000000000000 P a36d6431a73943f894c8e9b8ae4da145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a36d6431a73943f894c8e9b8ae4da145" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44647 } }
I20260928 08:49:06.128571 20960 raft_consensus.cc:388] T 00000000000000000000000000000000 P a36d6431a73943f894c8e9b8ae4da145 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:49:06.128737 20960 raft_consensus.cc:482] T 00000000000000000000000000000000 P a36d6431a73943f894c8e9b8ae4da145 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:49:06.128921 20960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a36d6431a73943f894c8e9b8ae4da145 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:49:06.131745 20960 raft_consensus.cc:504] T 00000000000000000000000000000000 P a36d6431a73943f894c8e9b8ae4da145 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a36d6431a73943f894c8e9b8ae4da145" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44647 } }
I20260928 08:49:06.132433 20960 leader_election.cc:302] T 00000000000000000000000000000000 P a36d6431a73943f894c8e9b8ae4da145 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a36d6431a73943f894c8e9b8ae4da145; no voters:
I20260928 08:49:06.133047 20960 leader_election.cc:288] T 00000000000000000000000000000000 P a36d6431a73943f894c8e9b8ae4da145 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:49:06.133222 20970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a36d6431a73943f894c8e9b8ae4da145 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:49:06.134881 20963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:06.126931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53581 (local address 127.0.0.1:44647)
0928 08:49:06.127127 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:06.127134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:06.127551 (+ 417us) server_negotiation.cc:407] Connection header received
0928 08:49:06.127788 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:06.127793 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:06.127895 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:06.128012 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:06.128513 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:06.129731 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:06.131068 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:06.131833 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:06.131962 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:06.133267 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:06.133274 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:06.133280 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:49:06.133305 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:06.133355 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:49:06.133382 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:06.134116 (+ 734us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:06.134189 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 08:49:06.134248 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 08:49:06.134967 20970 raft_consensus.cc:686] T 00000000000000000000000000000000 P a36d6431a73943f894c8e9b8ae4da145 [term 1 LEADER]: Becoming Leader. State: Replica: a36d6431a73943f894c8e9b8ae4da145, State: Running, Role: LEADER
I20260928 08:49:06.135856 20960 sys_catalog.cc:564] T 00000000000000000000000000000000 P a36d6431a73943f894c8e9b8ae4da145 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:49:06.136104 20970 consensus_queue.cc:237] T 00000000000000000000000000000000 P a36d6431a73943f894c8e9b8ae4da145 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a36d6431a73943f894c8e9b8ae4da145" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44647 } }
I20260928 08:49:06.139338 20972 sys_catalog.cc:455] T 00000000000000000000000000000000 P a36d6431a73943f894c8e9b8ae4da145 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a36d6431a73943f894c8e9b8ae4da145. Latest consensus state: current_term: 1 leader_uuid: "a36d6431a73943f894c8e9b8ae4da145" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a36d6431a73943f894c8e9b8ae4da145" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44647 } } }
I20260928 08:49:06.139367 20971 sys_catalog.cc:455] T 00000000000000000000000000000000 P a36d6431a73943f894c8e9b8ae4da145 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a36d6431a73943f894c8e9b8ae4da145" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a36d6431a73943f894c8e9b8ae4da145" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44647 } } }
I20260928 08:49:06.139997 20971 sys_catalog.cc:458] T 00000000000000000000000000000000 P a36d6431a73943f894c8e9b8ae4da145 [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:06.140005 20972 sys_catalog.cc:458] T 00000000000000000000000000000000 P a36d6431a73943f894c8e9b8ae4da145 [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:06.141618 20977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:49:06.147189 20977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:49:06.157368 20977 catalog_manager.cc:1347] Generated new cluster ID: 3640df2288ff460490913649033a774f
I20260928 08:49:06.157506 20977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:49:06.170912 20977 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:49:06.172281 20977 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:49:06.179958 20977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a36d6431a73943f894c8e9b8ae4da145: Generated new TSK 0
I20260928 08:49:06.180828 20977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:49:06.190246 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 20893
W20260928 08:49:06.191447 20955 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46597
W20260928 08:49:06.192435 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 20893: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:49:06.192619 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 20893
W20260928 08:49:06.209906 20878 connection.cc:535] client connection to 127.0.0.1:44647 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44647): Connection reset by peer (error 104)
2026-09-28T08:49:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1942 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:49:06 dist-test-slave-n6xs krb5kdc[20991](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:49:06 dist-test-slave-n6xs krb5kdc[20991](info): set up 2 sockets
Sep 28 08:49:06 dist-test-slave-n6xs krb5kdc[20991](info): commencing operation
krb5kdc: starting...
W20260928 08:49:07.296355 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:49:07 dist-test-slave-n6xs krb5kdc[20991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:49:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:49:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:49:07.563196 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33115
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:49:07.897799 21004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:49:07.898114 21004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:49:07.898274 21004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:49:07.905745 21004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:49:07.905877 21004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:49:07.906008 21004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:49:07.906113 21004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:49:07.906198 21004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:49:07.916388 21004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33115
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:49:07.917773 21004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:49:07.919131 21004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:49:07.931222 21012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:07.931298 21009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:07.931321 21010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:49:07.932767 21004 server_base.cc:1034] running on GCE node
I20260928 08:49:07.933571 21004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:49:07.935067 21004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:49:07.936299 21004 hybrid_clock.cc:648] HybridClock initialized: now 1790585347936259 us; error 64 us; skew 500 ppm
I20260928 08:49:07.936681 21004 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:49:07 dist-test-slave-n6xs krb5kdc[20991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:49:07.939920 21004 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:49:07.941447 21004 webserver.cc:460] Webserver started at http://127.0.0.1:45579/ using document root <none> and password file <none>
I20260928 08:49:07.942284 21004 fs_manager.cc:359] Metadata directory not provided
I20260928 08:49:07.942447 21004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:49:07.942760 21004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:49:07.945938 21004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "42ee65519ceb4979a82b53745753bd45"
format_stamp: "Formatted at 2026-09-28 08:49:07 on dist-test-slave-n6xs"
server_key: "d535d16d7d997c63ce0b5d9216cb952b"
server_key_iv: "453c3e1156eacbf7e5ca83ca198c6ca2"
server_key_version: "encryptionkey@0"
I20260928 08:49:07.946743 21004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "42ee65519ceb4979a82b53745753bd45"
format_stamp: "Formatted at 2026-09-28 08:49:07 on dist-test-slave-n6xs"
server_key: "d535d16d7d997c63ce0b5d9216cb952b"
server_key_iv: "453c3e1156eacbf7e5ca83ca198c6ca2"
server_key_version: "encryptionkey@0"
I20260928 08:49:07.952338 21004 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 08:49:07.956239 21019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:49:07.957793 21004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:49:07.958041 21004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "42ee65519ceb4979a82b53745753bd45"
format_stamp: "Formatted at 2026-09-28 08:49:07 on dist-test-slave-n6xs"
server_key: "d535d16d7d997c63ce0b5d9216cb952b"
server_key_iv: "453c3e1156eacbf7e5ca83ca198c6ca2"
server_key_version: "encryptionkey@0"
I20260928 08:49:07.958261 21004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:49:07.964555 21004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:49:07.968811 21004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:49:07.969107 21004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:49:07.988322 21004 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33115
I20260928 08:49:07.988448 21070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33115 every 8 connection(s)
I20260928 08:49:07.989909 21004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:49:07.993546 21071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:49:07.998802 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 21004
I20260928 08:49:07.998980 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:49:07.999472 388 external_mini_cluster.cc:1428] Setting key ff1ffb4757b35649e42177b83ce1bf01
I20260928 08:49:08.004830 21071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42ee65519ceb4979a82b53745753bd45: Bootstrap starting.
I20260928 08:49:08.008041 21074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:08.001414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46651 (local address 127.0.0.1:33115)
0928 08:49:08.002255 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:08.002272 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:08.002317 (+ 45us) server_negotiation.cc:407] Connection header received
0928 08:49:08.004312 (+ 1995us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:08.004338 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:08.004708 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:08.005116 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:08.006315 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:08.006322 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:08.006335 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:49:08.006351 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:08.006416 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:49:08.006442 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:08.006646 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:08.006820 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 08:49:08.007045 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":391,"threads_started":1}
I20260928 08:49:08.008953 21071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42ee65519ceb4979a82b53745753bd45: Neither blocks nor log segments found. Creating new log.
I20260928 08:49:08.009984 21071 log.cc:824] T 00000000000000000000000000000000 P 42ee65519ceb4979a82b53745753bd45: Log is configured to *not* fsync() on all Append() calls
I20260928 08:49:08.011909 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:49:08.013289 21071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42ee65519ceb4979a82b53745753bd45: No bootstrap required, opened a new log
I20260928 08:49:08.019686 21074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:08.017151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45837 (local address 127.0.0.1:33115)
0928 08:49:08.017375 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:08.017384 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:08.017695 (+ 311us) server_negotiation.cc:407] Connection header received
0928 08:49:08.017842 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:08.017845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:08.017901 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:08.017985 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:08.018958 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:08.018962 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:08.018966 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:49:08.018982 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:08.019033 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:49:08.019055 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:08.019171 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:08.019297 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 08:49:08.019346 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 08:49:08.022948 21071 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42ee65519ceb4979a82b53745753bd45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42ee65519ceb4979a82b53745753bd45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33115 } }
I20260928 08:49:08.023293 21071 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42ee65519ceb4979a82b53745753bd45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:49:08.023428 21071 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42ee65519ceb4979a82b53745753bd45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42ee65519ceb4979a82b53745753bd45, State: Initialized, Role: FOLLOWER
I20260928 08:49:08.024058 21071 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42ee65519ceb4979a82b53745753bd45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42ee65519ceb4979a82b53745753bd45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33115 } }
I20260928 08:49:08.024322 21071 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42ee65519ceb4979a82b53745753bd45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:49:08.024475 21071 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42ee65519ceb4979a82b53745753bd45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:49:08.024634 21071 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42ee65519ceb4979a82b53745753bd45 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:49:08.027050 21071 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42ee65519ceb4979a82b53745753bd45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42ee65519ceb4979a82b53745753bd45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33115 } }
I20260928 08:49:08.027535 21071 leader_election.cc:302] T 00000000000000000000000000000000 P 42ee65519ceb4979a82b53745753bd45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42ee65519ceb4979a82b53745753bd45; no voters:
I20260928 08:49:08.028148 21071 leader_election.cc:288] T 00000000000000000000000000000000 P 42ee65519ceb4979a82b53745753bd45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:49:08.028297 21081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42ee65519ceb4979a82b53745753bd45 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:49:08.030004 21081 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42ee65519ceb4979a82b53745753bd45 [term 1 LEADER]: Becoming Leader. State: Replica: 42ee65519ceb4979a82b53745753bd45, State: Running, Role: LEADER
I20260928 08:49:08.030224 21071 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42ee65519ceb4979a82b53745753bd45 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:49:08.030543 21081 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42ee65519ceb4979a82b53745753bd45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42ee65519ceb4979a82b53745753bd45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33115 } }
I20260928 08:49:08.033361 21083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42ee65519ceb4979a82b53745753bd45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42ee65519ceb4979a82b53745753bd45. Latest consensus state: current_term: 1 leader_uuid: "42ee65519ceb4979a82b53745753bd45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42ee65519ceb4979a82b53745753bd45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33115 } } }
I20260928 08:49:08.033334 21082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42ee65519ceb4979a82b53745753bd45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42ee65519ceb4979a82b53745753bd45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42ee65519ceb4979a82b53745753bd45" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33115 } } }
I20260928 08:49:08.033864 21082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42ee65519ceb4979a82b53745753bd45 [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:08.033864 21083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42ee65519ceb4979a82b53745753bd45 [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:08.036164 21090 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:49:08.041277 21090 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:49:08.050726 21090 catalog_manager.cc:1347] Generated new cluster ID: fd4aa33cdf704c34a169b14373ba5d5d
I20260928 08:49:08.050853 21090 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:49:08.064898 21090 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:49:08.065413 21090 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:49:08.078734 21090 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42ee65519ceb4979a82b53745753bd45: Generated new TSK 0
I20260928 08:49:08.079628 21090 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:49:08.088752 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 21004
W20260928 08:49:08.089874 21066 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46651
W20260928 08:49:08.090584 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 21004: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:49:08.090735 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 21004
2026-09-28T08:49:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1897 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:49:08 dist-test-slave-n6xs krb5kdc[21102](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:49:08 dist-test-slave-n6xs krb5kdc[21102](info): set up 2 sockets
Sep 28 08:49:08 dist-test-slave-n6xs krb5kdc[21102](info): commencing operation
krb5kdc: starting...
W20260928 08:49:09.212051 388 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:49:09 dist-test-slave-n6xs krb5kdc[21102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:49:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:49:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:49:09.482853 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43759
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:49:09.825930 21115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:49:09.826319 21115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:49:09.826488 21115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:49:09.833861 21115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:49:09.834059 21115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:49:09.834151 21115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:49:09.834237 21115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:49:09.834344 21115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:49:09.844551 21115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:49:09.846029 21115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:49:09.847365 21115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:49:09.860013 21123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:09.860044 21120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:09.859967 21121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:49:09.860608 21115 server_base.cc:1034] running on GCE node
I20260928 08:49:09.861557 21115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:49:09.863040 21115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:49:09.864252 21115 hybrid_clock.cc:648] HybridClock initialized: now 1790585349864167 us; error 74 us; skew 500 ppm
I20260928 08:49:09.864683 21115 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:49:09 dist-test-slave-n6xs krb5kdc[21102](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:49:09.868142 21115 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:49:09.869777 21115 webserver.cc:460] Webserver started at http://127.0.0.1:42453/ using document root <none> and password file <none>
I20260928 08:49:09.870498 21115 fs_manager.cc:359] Metadata directory not provided
I20260928 08:49:09.870606 21115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:49:09.870865 21115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:49:09.874114 21115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "13532cb1767f44a293ad57985fed163a"
format_stamp: "Formatted at 2026-09-28 08:49:09 on dist-test-slave-n6xs"
server_key: "647a774b282267b5d5d4fbacaf3b72d0"
server_key_iv: "f2af4c040a69a4a873aece4bfdac09ad"
server_key_version: "encryptionkey@0"
I20260928 08:49:09.874878 21115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "13532cb1767f44a293ad57985fed163a"
format_stamp: "Formatted at 2026-09-28 08:49:09 on dist-test-slave-n6xs"
server_key: "647a774b282267b5d5d4fbacaf3b72d0"
server_key_iv: "f2af4c040a69a4a873aece4bfdac09ad"
server_key_version: "encryptionkey@0"
I20260928 08:49:09.880679 21115 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 08:49:09.884756 21130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:49:09.886343 21115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 08:49:09.886585 21115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "13532cb1767f44a293ad57985fed163a"
format_stamp: "Formatted at 2026-09-28 08:49:09 on dist-test-slave-n6xs"
server_key: "647a774b282267b5d5d4fbacaf3b72d0"
server_key_iv: "f2af4c040a69a4a873aece4bfdac09ad"
server_key_version: "encryptionkey@0"
I20260928 08:49:09.886838 21115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:49:09.908613 21115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:49:09.913300 21115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:49:09.913626 21115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:49:09.934283 21115 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43759
I20260928 08:49:09.934315 21181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43759 every 8 connection(s)
I20260928 08:49:09.935911 21115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:49:09.939240 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 21115
I20260928 08:49:09.939422 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:49:09.939951 388 external_mini_cluster.cc:1428] Setting key 4e505d6102084d9ffffed186851158fa
I20260928 08:49:09.940186 21182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:49:09.953961 21182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13532cb1767f44a293ad57985fed163a: Bootstrap starting.
Sep 28 08:49:09 dist-test-slave-n6xs krb5kdc[21102](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 08:49:09.957665 21182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13532cb1767f44a293ad57985fed163a: Neither blocks nor log segments found. Creating new log.
I20260928 08:49:09.958905 21182 log.cc:824] T 00000000000000000000000000000000 P 13532cb1767f44a293ad57985fed163a: Log is configured to *not* fsync() on all Append() calls
I20260928 08:49:09.962077 21182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13532cb1767f44a293ad57985fed163a: No bootstrap required, opened a new log
I20260928 08:49:09.965144 21185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:09.941770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36393 (local address 127.0.0.1:43759)
0928 08:49:09.942684 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:09.942702 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:09.943000 (+ 298us) server_negotiation.cc:407] Connection header received
0928 08:49:09.945538 (+ 2538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:09.945566 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:09.946072 (+ 506us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:09.946550 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:09.948257 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:09.949766 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:09.951125 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:09.952022 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:09.952276 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:09.956969 (+ 4693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:09.956979 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:09.956996 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:09.957045 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:09.960571 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:09.961425 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:09.961433 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:09.961439 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:09.961619 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:09.962065 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:09.962069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:09.962071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:09.962684 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:09.963018 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:09.963482 (+ 464us) server_negotiation.cc:299] Negotiation successful
0928 08:49:09.964032 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":395,"threads_started":1}
I20260928 08:49:09.969192 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:49:09.970600 21182 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13532cb1767f44a293ad57985fed163a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13532cb1767f44a293ad57985fed163a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43759 } }
I20260928 08:49:09.970999 21182 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13532cb1767f44a293ad57985fed163a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:49:09.971117 21182 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13532cb1767f44a293ad57985fed163a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13532cb1767f44a293ad57985fed163a, State: Initialized, Role: FOLLOWER
I20260928 08:49:09.971900 21182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13532cb1767f44a293ad57985fed163a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13532cb1767f44a293ad57985fed163a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43759 } }
I20260928 08:49:09.972184 21182 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13532cb1767f44a293ad57985fed163a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:49:09.972319 21182 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13532cb1767f44a293ad57985fed163a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:49:09.972506 21182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13532cb1767f44a293ad57985fed163a [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:49:09.975386 21182 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13532cb1767f44a293ad57985fed163a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13532cb1767f44a293ad57985fed163a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43759 } }
I20260928 08:49:09.976090 21182 leader_election.cc:302] T 00000000000000000000000000000000 P 13532cb1767f44a293ad57985fed163a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13532cb1767f44a293ad57985fed163a; no voters:
I20260928 08:49:09.976828 21182 leader_election.cc:288] T 00000000000000000000000000000000 P 13532cb1767f44a293ad57985fed163a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:49:09.976945 21192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13532cb1767f44a293ad57985fed163a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:49:09.979640 21182 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13532cb1767f44a293ad57985fed163a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:49:09.983886 21192 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13532cb1767f44a293ad57985fed163a [term 1 LEADER]: Becoming Leader. State: Replica: 13532cb1767f44a293ad57985fed163a, State: Running, Role: LEADER
I20260928 08:49:09.984808 21192 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13532cb1767f44a293ad57985fed163a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13532cb1767f44a293ad57985fed163a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43759 } }
I20260928 08:49:09.988601 21193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13532cb1767f44a293ad57985fed163a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13532cb1767f44a293ad57985fed163a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13532cb1767f44a293ad57985fed163a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43759 } } }
I20260928 08:49:09.989125 21193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13532cb1767f44a293ad57985fed163a [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:09.990067 21194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13532cb1767f44a293ad57985fed163a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13532cb1767f44a293ad57985fed163a. Latest consensus state: current_term: 1 leader_uuid: "13532cb1767f44a293ad57985fed163a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13532cb1767f44a293ad57985fed163a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43759 } } }
I20260928 08:49:09.990480 21194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13532cb1767f44a293ad57985fed163a [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:09.990490 21185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:09.973805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58871 (local address 127.0.0.1:43759)
0928 08:49:09.974004 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:09.974011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:09.974348 (+ 337us) server_negotiation.cc:407] Connection header received
0928 08:49:09.974560 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:09.974566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:09.974646 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:09.974749 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:09.976284 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:09.977156 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:09.978744 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:09.979274 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:09.979422 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:09.982271 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:09.982278 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:09.982282 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:09.982323 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:09.986008 (+ 3685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:09.986765 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:09.986771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:09.986775 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:09.986938 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:09.987484 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:09.987489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:09.987492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:09.988088 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:09.989411 (+ 1323us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:09.989626 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 08:49:09.990036 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 08:49:09.997946 21201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:49:10.004009 21201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:49:10.014175 21201 catalog_manager.cc:1347] Generated new cluster ID: 393c0d7a54034fe989784e3d02245f7a
I20260928 08:49:10.014322 21201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:49:10.040994 21201 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:49:10.042454 21201 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:49:10.057708 21201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13532cb1767f44a293ad57985fed163a: Generated new TSK 0
I20260928 08:49:10.058671 21201 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:49:10.069329 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 21115
I20260928 08:49:10.070605 21177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:49:10.167361 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 21115
2026-09-28T08:49:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2077 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:49:10 dist-test-slave-n6xs krb5kdc[21215](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:49:10 dist-test-slave-n6xs krb5kdc[21215](info): set up 2 sockets
Sep 28 08:49:10 dist-test-slave-n6xs krb5kdc[21215](info): commencing operation
krb5kdc: starting...
W20260928 08:49:11.280331 388 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:49:11 dist-test-slave-n6xs krb5kdc[21215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:49:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:49:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:49:11.547086 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33531
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:49:11.881705 21228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:49:11.882036 21228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:49:11.882215 21228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:49:11.889720 21228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:49:11.889854 21228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:49:11.889955 21228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:49:11.890071 21228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:49:11.890137 21228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:49:11.900460 21228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:49:11.901909 21228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:49:11.903337 21228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:49:11.916702 21233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:11.916584 21236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:11.916574 21234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:49:11.918215 21228 server_base.cc:1034] running on GCE node
I20260928 08:49:11.919164 21228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:49:11.921060 21228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:49:11.922302 21228 hybrid_clock.cc:648] HybridClock initialized: now 1790585351922220 us; error 76 us; skew 500 ppm
I20260928 08:49:11.922726 21228 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:49:11 dist-test-slave-n6xs krb5kdc[21215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:49:11.930416 21228 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:49:11.932297 21228 webserver.cc:460] Webserver started at http://127.0.0.1:45781/ using document root <none> and password file <none>
I20260928 08:49:11.933106 21228 fs_manager.cc:359] Metadata directory not provided
I20260928 08:49:11.933248 21228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:49:11.933581 21228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:49:11.936961 21228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "86a77e1443514a3daf48ce162cb696e6"
format_stamp: "Formatted at 2026-09-28 08:49:11 on dist-test-slave-n6xs"
server_key: "ba5af1051fdb6f65b681208227da542e"
server_key_iv: "3e3a4c367c77a16babdf6998b4da82c0"
server_key_version: "encryptionkey@0"
I20260928 08:49:11.937805 21228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "86a77e1443514a3daf48ce162cb696e6"
format_stamp: "Formatted at 2026-09-28 08:49:11 on dist-test-slave-n6xs"
server_key: "ba5af1051fdb6f65b681208227da542e"
server_key_iv: "3e3a4c367c77a16babdf6998b4da82c0"
server_key_version: "encryptionkey@0"
I20260928 08:49:11.943645 21228 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 08:49:11.947788 21243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:49:11.949625 21228 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 08:49:11.949855 21228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "86a77e1443514a3daf48ce162cb696e6"
format_stamp: "Formatted at 2026-09-28 08:49:11 on dist-test-slave-n6xs"
server_key: "ba5af1051fdb6f65b681208227da542e"
server_key_iv: "3e3a4c367c77a16babdf6998b4da82c0"
server_key_version: "encryptionkey@0"
I20260928 08:49:11.950098 21228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:49:11.963686 21228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:49:11.968568 21228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:49:11.968891 21228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:49:11.988729 21228 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33531
I20260928 08:49:11.988768 21294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33531 every 8 connection(s)
I20260928 08:49:11.990217 21228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:49:11.992525 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 21228
I20260928 08:49:11.992722 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:49:11.993180 388 external_mini_cluster.cc:1428] Setting key 9070db2f35f1454f9cab0aa80df07e04
I20260928 08:49:11.994962 21295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:49:12.010063 21298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:11.995112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38985 (local address 127.0.0.1:33531)
0928 08:49:11.995973 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:11.995992 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:11.996050 (+ 58us) server_negotiation.cc:407] Connection header received
0928 08:49:11.998443 (+ 2393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:11.998475 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:11.999085 (+ 610us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:11.999612 (+ 527us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:12.000217 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:12.002520 (+ 2303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:12.003998 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:12.005204 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:12.005420 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:12.006571 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:12.006584 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:12.006605 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:49:12.007467 (+ 862us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:12.007581 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:49:12.007628 (+ 47us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:12.007769 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:12.008094 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 08:49:12.008435 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":338,"threads_started":1}
I20260928 08:49:12.011621 21295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86a77e1443514a3daf48ce162cb696e6: Bootstrap starting.
I20260928 08:49:12.014196 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:49:12.015704 21295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86a77e1443514a3daf48ce162cb696e6: Neither blocks nor log segments found. Creating new log.
I20260928 08:49:12.017185 21295 log.cc:824] T 00000000000000000000000000000000 P 86a77e1443514a3daf48ce162cb696e6: Log is configured to *not* fsync() on all Append() calls
I20260928 08:49:12.020545 21295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86a77e1443514a3daf48ce162cb696e6: No bootstrap required, opened a new log
I20260928 08:49:12.027714 21298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:12.020377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46233 (local address 127.0.0.1:33531)
0928 08:49:12.020593 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:12.020602 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:12.021080 (+ 478us) server_negotiation.cc:407] Connection header received
0928 08:49:12.021259 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:12.021264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:12.021416 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:12.021552 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:12.022237 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:12.023410 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:12.024907 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:12.025624 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:12.025758 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:12.026923 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:12.026928 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:12.026931 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:49:12.026954 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:12.026994 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:49:12.027010 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:12.027124 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:12.027271 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 08:49:12.027390 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 08:49:12.030496 21295 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86a77e1443514a3daf48ce162cb696e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a77e1443514a3daf48ce162cb696e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33531 } }
I20260928 08:49:12.030925 21295 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86a77e1443514a3daf48ce162cb696e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:49:12.031040 21295 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86a77e1443514a3daf48ce162cb696e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a77e1443514a3daf48ce162cb696e6, State: Initialized, Role: FOLLOWER
I20260928 08:49:12.031621 21295 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86a77e1443514a3daf48ce162cb696e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a77e1443514a3daf48ce162cb696e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33531 } }
I20260928 08:49:12.031900 21295 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86a77e1443514a3daf48ce162cb696e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:49:12.032065 21295 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86a77e1443514a3daf48ce162cb696e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:49:12.032245 21295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86a77e1443514a3daf48ce162cb696e6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:49:12.034783 21295 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86a77e1443514a3daf48ce162cb696e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a77e1443514a3daf48ce162cb696e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33531 } }
I20260928 08:49:12.035311 21295 leader_election.cc:302] T 00000000000000000000000000000000 P 86a77e1443514a3daf48ce162cb696e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86a77e1443514a3daf48ce162cb696e6; no voters:
I20260928 08:49:12.035992 21295 leader_election.cc:288] T 00000000000000000000000000000000 P 86a77e1443514a3daf48ce162cb696e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:49:12.036130 21305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86a77e1443514a3daf48ce162cb696e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:49:12.036868 21305 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86a77e1443514a3daf48ce162cb696e6 [term 1 LEADER]: Becoming Leader. State: Replica: 86a77e1443514a3daf48ce162cb696e6, State: Running, Role: LEADER
I20260928 08:49:12.037442 21305 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86a77e1443514a3daf48ce162cb696e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a77e1443514a3daf48ce162cb696e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33531 } }
I20260928 08:49:12.038259 21295 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86a77e1443514a3daf48ce162cb696e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:49:12.041905 21306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86a77e1443514a3daf48ce162cb696e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86a77e1443514a3daf48ce162cb696e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a77e1443514a3daf48ce162cb696e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33531 } } }
I20260928 08:49:12.041993 21307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86a77e1443514a3daf48ce162cb696e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86a77e1443514a3daf48ce162cb696e6. Latest consensus state: current_term: 1 leader_uuid: "86a77e1443514a3daf48ce162cb696e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86a77e1443514a3daf48ce162cb696e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33531 } } }
I20260928 08:49:12.042323 21306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86a77e1443514a3daf48ce162cb696e6 [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:12.042395 21307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86a77e1443514a3daf48ce162cb696e6 [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:12.044063 21313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:49:12.049856 21313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:49:12.060278 21313 catalog_manager.cc:1347] Generated new cluster ID: e70e33c2033744f5ba1d1e0de0cc9157
I20260928 08:49:12.060447 21313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:49:12.071228 21313 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:49:12.072748 21313 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:49:12.078187 21313 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86a77e1443514a3daf48ce162cb696e6: Generated new TSK 0
I20260928 08:49:12.079331 21313 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:49:12.093957 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 21228
W20260928 08:49:12.095193 21290 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38985
W20260928 08:49:12.095956 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 21228: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:49:12.096112 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 21228
2026-09-28T08:49:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1928 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:49:12 dist-test-slave-n6xs krb5kdc[21326](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:49:12 dist-test-slave-n6xs krb5kdc[21326](info): set up 2 sockets
Sep 28 08:49:12 dist-test-slave-n6xs krb5kdc[21326](info): commencing operation
krb5kdc: starting...
W20260928 08:49:13.192222 388 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:49:13 dist-test-slave-n6xs krb5kdc[21326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:49:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:49:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:49:13.440982 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35051
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:49:13.758829 21339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:49:13.759107 21339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:49:13.759303 21339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:49:13.766472 21339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:49:13.766579 21339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:49:13.766682 21339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:49:13.766785 21339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:49:13.766847 21339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:49:13.776860 21339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:49:13.778194 21339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:49:13.779527 21339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:49:13.791136 21344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:13.791175 21345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:13.791290 21347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:49:13.792300 21339 server_base.cc:1034] running on GCE node
I20260928 08:49:13.793073 21339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:49:13.794337 21339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:49:13.795543 21339 hybrid_clock.cc:648] HybridClock initialized: now 1790585353795507 us; error 25 us; skew 500 ppm
I20260928 08:49:13.795912 21339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:49:13 dist-test-slave-n6xs krb5kdc[21326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:49:13.798954 21339 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:49:13.800446 21339 webserver.cc:460] Webserver started at http://127.0.0.1:40019/ using document root <none> and password file <none>
I20260928 08:49:13.801069 21339 fs_manager.cc:359] Metadata directory not provided
I20260928 08:49:13.801164 21339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:49:13.801407 21339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:49:13.804510 21339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "6216d35450f94a0cb14e0e5ded8022bc"
format_stamp: "Formatted at 2026-09-28 08:49:13 on dist-test-slave-n6xs"
server_key: "8fb3f23b6ead37784bbac87ef55d2c2d"
server_key_iv: "5b4a4367d49eb95676f35050e4ec69be"
server_key_version: "encryptionkey@0"
I20260928 08:49:13.805222 21339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "6216d35450f94a0cb14e0e5ded8022bc"
format_stamp: "Formatted at 2026-09-28 08:49:13 on dist-test-slave-n6xs"
server_key: "8fb3f23b6ead37784bbac87ef55d2c2d"
server_key_iv: "5b4a4367d49eb95676f35050e4ec69be"
server_key_version: "encryptionkey@0"
I20260928 08:49:13.810232 21339 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:49:13.813663 21354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:49:13.814947 21339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:49:13.815083 21339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "6216d35450f94a0cb14e0e5ded8022bc"
format_stamp: "Formatted at 2026-09-28 08:49:13 on dist-test-slave-n6xs"
server_key: "8fb3f23b6ead37784bbac87ef55d2c2d"
server_key_iv: "5b4a4367d49eb95676f35050e4ec69be"
server_key_version: "encryptionkey@0"
I20260928 08:49:13.815237 21339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:49:13.820577 21339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:49:13.824790 21339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:49:13.825058 21339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:49:13.843683 21339 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35051
I20260928 08:49:13.843711 21405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35051 every 8 connection(s)
I20260928 08:49:13.845127 21339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:49:13.848968 21406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:49:13.853168 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 21339
I20260928 08:49:13.853391 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:49:13.853832 388 external_mini_cluster.cc:1428] Setting key a599d81144871d526190e254df770607
I20260928 08:49:13.859721 21406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6216d35450f94a0cb14e0e5ded8022bc: Bootstrap starting.
I20260928 08:49:13.861913 21409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:13.855602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54293 (local address 127.0.0.1:35051)
0928 08:49:13.856488 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:13.856502 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:13.856543 (+ 41us) server_negotiation.cc:407] Connection header received
0928 08:49:13.858512 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:13.858535 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:13.858865 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:13.859182 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:13.860302 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:13.860309 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:13.860321 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:49:13.860349 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:13.860406 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:49:13.860426 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:13.860590 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:13.860738 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 08:49:13.860950 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":452,"threads_started":1}
I20260928 08:49:13.863363 21406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6216d35450f94a0cb14e0e5ded8022bc: Neither blocks nor log segments found. Creating new log.
I20260928 08:49:13.864559 21406 log.cc:824] T 00000000000000000000000000000000 P 6216d35450f94a0cb14e0e5ded8022bc: Log is configured to *not* fsync() on all Append() calls
I20260928 08:49:13.865998 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:49:13.867563 21406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6216d35450f94a0cb14e0e5ded8022bc: No bootstrap required, opened a new log
I20260928 08:49:13.873376 21409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:13.870964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39371 (local address 127.0.0.1:35051)
0928 08:49:13.871139 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:13.871144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:13.871398 (+ 254us) server_negotiation.cc:407] Connection header received
0928 08:49:13.871584 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:13.871588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:13.871641 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:13.871719 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:13.872658 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:13.872663 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:13.872666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:49:13.872679 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:13.872717 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:49:13.872731 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:13.872820 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:13.873014 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 08:49:13.873105 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 08:49:13.876132 21406 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6216d35450f94a0cb14e0e5ded8022bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6216d35450f94a0cb14e0e5ded8022bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35051 } }
I20260928 08:49:13.876518 21406 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6216d35450f94a0cb14e0e5ded8022bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:49:13.876631 21406 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6216d35450f94a0cb14e0e5ded8022bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6216d35450f94a0cb14e0e5ded8022bc, State: Initialized, Role: FOLLOWER
I20260928 08:49:13.877283 21406 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6216d35450f94a0cb14e0e5ded8022bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6216d35450f94a0cb14e0e5ded8022bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35051 } }
I20260928 08:49:13.877545 21406 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6216d35450f94a0cb14e0e5ded8022bc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:49:13.877683 21406 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6216d35450f94a0cb14e0e5ded8022bc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:49:13.877871 21406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6216d35450f94a0cb14e0e5ded8022bc [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:49:13.880038 21406 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6216d35450f94a0cb14e0e5ded8022bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6216d35450f94a0cb14e0e5ded8022bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35051 } }
I20260928 08:49:13.880506 21406 leader_election.cc:302] T 00000000000000000000000000000000 P 6216d35450f94a0cb14e0e5ded8022bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6216d35450f94a0cb14e0e5ded8022bc; no voters:
I20260928 08:49:13.881068 21406 leader_election.cc:288] T 00000000000000000000000000000000 P 6216d35450f94a0cb14e0e5ded8022bc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:49:13.881196 21416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6216d35450f94a0cb14e0e5ded8022bc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:49:13.881896 21416 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6216d35450f94a0cb14e0e5ded8022bc [term 1 LEADER]: Becoming Leader. State: Replica: 6216d35450f94a0cb14e0e5ded8022bc, State: Running, Role: LEADER
I20260928 08:49:13.882409 21416 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6216d35450f94a0cb14e0e5ded8022bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6216d35450f94a0cb14e0e5ded8022bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35051 } }
I20260928 08:49:13.883105 21406 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6216d35450f94a0cb14e0e5ded8022bc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:49:13.885342 21418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6216d35450f94a0cb14e0e5ded8022bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6216d35450f94a0cb14e0e5ded8022bc. Latest consensus state: current_term: 1 leader_uuid: "6216d35450f94a0cb14e0e5ded8022bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6216d35450f94a0cb14e0e5ded8022bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35051 } } }
I20260928 08:49:13.885438 21417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6216d35450f94a0cb14e0e5ded8022bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6216d35450f94a0cb14e0e5ded8022bc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6216d35450f94a0cb14e0e5ded8022bc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35051 } } }
I20260928 08:49:13.885845 21418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6216d35450f94a0cb14e0e5ded8022bc [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:13.885914 21417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6216d35450f94a0cb14e0e5ded8022bc [sys.catalog]: This master's current role is: LEADER
W20260928 08:49:13.894672 21431 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6216d35450f94a0cb14e0e5ded8022bc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:49:13.894837 21431 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:49:13.895067 21423 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:49:13.896759 21423 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:49:13.904443 21423 catalog_manager.cc:1347] Generated new cluster ID: 57b2cdeff89046ca9fcf49039203b21f
I20260928 08:49:13.904549 21423 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:49:13.919030 21423 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:49:13.919517 21423 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:49:13.926035 21423 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6216d35450f94a0cb14e0e5ded8022bc: Generated new TSK 0
I20260928 08:49:13.926929 21423 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:49:13.940168 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 21339
W20260928 08:49:13.941385 21401 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54293
W20260928 08:49:13.942052 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 21339: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:49:13.942201 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 21339
2026-09-28T08:49:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1845 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:49:14 dist-test-slave-n6xs krb5kdc[21437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:49:14 dist-test-slave-n6xs krb5kdc[21437](info): set up 2 sockets
Sep 28 08:49:14 dist-test-slave-n6xs krb5kdc[21437](info): commencing operation
krb5kdc: starting...
W20260928 08:49:15.028286 388 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:49:15 dist-test-slave-n6xs krb5kdc[21437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:49:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:49:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:49:15.279345 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40161
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:49:15.615363 21450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:49:15.615664 21450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:49:15.615788 21450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:49:15.623209 21450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:49:15.623328 21450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:49:15.623389 21450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:49:15.623458 21450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:49:15.623508 21450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:49:15.633795 21450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:49:15.635073 21450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:49:15.636385 21450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:49:15.648794 21455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:15.648870 21456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:15.648940 21458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:49:15.650259 21450 server_base.cc:1034] running on GCE node
I20260928 08:49:15.650962 21450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:49:15.652408 21450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:49:15.653656 21450 hybrid_clock.cc:648] HybridClock initialized: now 1790585355653597 us; error 47 us; skew 500 ppm
I20260928 08:49:15.654008 21450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:49:15 dist-test-slave-n6xs krb5kdc[21437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:49:15.657064 21450 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:49:15.658456 21450 webserver.cc:460] Webserver started at http://127.0.0.1:43197/ using document root <none> and password file <none>
I20260928 08:49:15.659125 21450 fs_manager.cc:359] Metadata directory not provided
I20260928 08:49:15.659260 21450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:49:15.659579 21450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:49:15.662649 21450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "b4e0c9c2bf8b4de1bf85c2a8ebb9ef47"
format_stamp: "Formatted at 2026-09-28 08:49:15 on dist-test-slave-n6xs"
server_key: "9f7ee5a65fe67a83fbc786d3985a8333"
server_key_iv: "93780a4a46bccb910010b4eee307acb1"
server_key_version: "encryptionkey@0"
I20260928 08:49:15.663403 21450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "b4e0c9c2bf8b4de1bf85c2a8ebb9ef47"
format_stamp: "Formatted at 2026-09-28 08:49:15 on dist-test-slave-n6xs"
server_key: "9f7ee5a65fe67a83fbc786d3985a8333"
server_key_iv: "93780a4a46bccb910010b4eee307acb1"
server_key_version: "encryptionkey@0"
I20260928 08:49:15.668455 21450 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 08:49:15.671782 21465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:49:15.673203 21450 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:49:15.673384 21450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "b4e0c9c2bf8b4de1bf85c2a8ebb9ef47"
format_stamp: "Formatted at 2026-09-28 08:49:15 on dist-test-slave-n6xs"
server_key: "9f7ee5a65fe67a83fbc786d3985a8333"
server_key_iv: "93780a4a46bccb910010b4eee307acb1"
server_key_version: "encryptionkey@0"
I20260928 08:49:15.673576 21450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:49:15.701116 21450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:49:15.705170 21450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:49:15.705556 21450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:49:15.723681 21450 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40161
I20260928 08:49:15.723718 21516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40161 every 8 connection(s)
I20260928 08:49:15.725028 21450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:49:15.728669 21517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:49:15.732834 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 21450
I20260928 08:49:15.733029 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:49:15.733536 388 external_mini_cluster.cc:1428] Setting key b554cf8c75cc50a9d1edacf9b270a919
I20260928 08:49:15.741958 21517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4e0c9c2bf8b4de1bf85c2a8ebb9ef47: Bootstrap starting.
I20260928 08:49:15.745848 21517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4e0c9c2bf8b4de1bf85c2a8ebb9ef47: Neither blocks nor log segments found. Creating new log.
I20260928 08:49:15.747185 21517 log.cc:824] T 00000000000000000000000000000000 P b4e0c9c2bf8b4de1bf85c2a8ebb9ef47: Log is configured to *not* fsync() on all Append() calls
Sep 28 08:49:15 dist-test-slave-n6xs krb5kdc[21437](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 08:49:15.750034 21517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4e0c9c2bf8b4de1bf85c2a8ebb9ef47: No bootstrap required, opened a new log
I20260928 08:49:15.756933 21520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:15.735349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:36865 (local address 127.0.0.1:40161)
0928 08:49:15.736142 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:15.736152 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:15.736194 (+ 42us) server_negotiation.cc:407] Connection header received
0928 08:49:15.737827 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:15.737850 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:15.738296 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:15.738623 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:15.740415 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:15.741786 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:15.743634 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:15.744466 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:15.744606 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:15.749489 (+ 4883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:15.749497 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:15.749511 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:15.749546 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:15.752685 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:15.753505 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:15.753530 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:15.753539 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:15.753701 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:15.754197 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:15.754201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:15.754203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:15.754735 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:15.755085 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:15.755577 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 08:49:15.755939 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":342,"threads_started":1}
I20260928 08:49:15.758481 21517 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4e0c9c2bf8b4de1bf85c2a8ebb9ef47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e0c9c2bf8b4de1bf85c2a8ebb9ef47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40161 } }
I20260928 08:49:15.758852 21517 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4e0c9c2bf8b4de1bf85c2a8ebb9ef47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:49:15.758975 21517 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4e0c9c2bf8b4de1bf85c2a8ebb9ef47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4e0c9c2bf8b4de1bf85c2a8ebb9ef47, State: Initialized, Role: FOLLOWER
I20260928 08:49:15.759648 21517 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4e0c9c2bf8b4de1bf85c2a8ebb9ef47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e0c9c2bf8b4de1bf85c2a8ebb9ef47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40161 } }
I20260928 08:49:15.759924 21517 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4e0c9c2bf8b4de1bf85c2a8ebb9ef47 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:49:15.760075 21517 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4e0c9c2bf8b4de1bf85c2a8ebb9ef47 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:49:15.760265 21517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4e0c9c2bf8b4de1bf85c2a8ebb9ef47 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:49:15.760932 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:49:15.762671 21517 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4e0c9c2bf8b4de1bf85c2a8ebb9ef47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e0c9c2bf8b4de1bf85c2a8ebb9ef47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40161 } }
I20260928 08:49:15.763152 21517 leader_election.cc:302] T 00000000000000000000000000000000 P b4e0c9c2bf8b4de1bf85c2a8ebb9ef47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4e0c9c2bf8b4de1bf85c2a8ebb9ef47; no voters:
I20260928 08:49:15.763778 21517 leader_election.cc:288] T 00000000000000000000000000000000 P b4e0c9c2bf8b4de1bf85c2a8ebb9ef47 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:49:15.763899 21523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4e0c9c2bf8b4de1bf85c2a8ebb9ef47 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:49:15.766284 21517 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4e0c9c2bf8b4de1bf85c2a8ebb9ef47 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:49:15.770864 21523 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4e0c9c2bf8b4de1bf85c2a8ebb9ef47 [term 1 LEADER]: Becoming Leader. State: Replica: b4e0c9c2bf8b4de1bf85c2a8ebb9ef47, State: Running, Role: LEADER
I20260928 08:49:15.771603 21523 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4e0c9c2bf8b4de1bf85c2a8ebb9ef47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e0c9c2bf8b4de1bf85c2a8ebb9ef47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40161 } }
I20260928 08:49:15.774891 21525 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4e0c9c2bf8b4de1bf85c2a8ebb9ef47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4e0c9c2bf8b4de1bf85c2a8ebb9ef47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e0c9c2bf8b4de1bf85c2a8ebb9ef47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40161 } } }
I20260928 08:49:15.775357 21525 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4e0c9c2bf8b4de1bf85c2a8ebb9ef47 [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:15.776075 21526 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4e0c9c2bf8b4de1bf85c2a8ebb9ef47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4e0c9c2bf8b4de1bf85c2a8ebb9ef47. Latest consensus state: current_term: 1 leader_uuid: "b4e0c9c2bf8b4de1bf85c2a8ebb9ef47" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e0c9c2bf8b4de1bf85c2a8ebb9ef47" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40161 } } }
I20260928 08:49:15.776425 21526 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4e0c9c2bf8b4de1bf85c2a8ebb9ef47 [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:15.797428 21537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:49:15.812302 21537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:49:15.821458 21520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:15.774470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:49515 (local address 127.0.0.1:40161)
0928 08:49:15.774625 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:15.774631 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:15.800750 (+ 26119us) server_negotiation.cc:407] Connection header received
0928 08:49:15.800984 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:15.800993 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:15.801116 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:15.801277 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:15.803919 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:15.805512 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:15.808132 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:15.809044 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:15.809170 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:15.813638 (+ 4468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:15.813646 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:15.813651 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:15.813693 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:15.818085 (+ 4392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:15.819097 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:15.819102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:15.819106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:15.819252 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:15.819764 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:15.819769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:15.819772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:15.820197 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:15.820521 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:15.820918 (+ 397us) server_negotiation.cc:299] Negotiation successful
0928 08:49:15.821085 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 08:49:15.825915 21537 catalog_manager.cc:1347] Generated new cluster ID: fe3ed39bc7284cd48dfe6df1958cb109
I20260928 08:49:15.826033 21537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:49:15.861173 21537 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:49:15.862655 21537 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:49:15.874536 21537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4e0c9c2bf8b4de1bf85c2a8ebb9ef47: Generated new TSK 0
I20260928 08:49:15.875669 21537 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:49:15.892424 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 21450
I20260928 08:49:15.893611 21512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:49:15.987653 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 21450
W20260928 08:49:16.006707 21435 connection.cc:535] client connection to 127.0.0.1:40161 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40161): Connection reset by peer (error 104)
2026-09-28T08:49:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2047 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:49:16 dist-test-slave-n6xs krb5kdc[21550](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:49:16 dist-test-slave-n6xs krb5kdc[21550](info): set up 2 sockets
Sep 28 08:49:16 dist-test-slave-n6xs krb5kdc[21550](info): commencing operation
krb5kdc: starting...
W20260928 08:49:17.080078 388 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:49:17 dist-test-slave-n6xs krb5kdc[21550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:49:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:49:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:49:17.330988 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41941
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46055
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:49:17.654626 21563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:49:17.654917 21563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:49:17.655056 21563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:49:17.662303 21563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:49:17.662420 21563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:49:17.662554 21563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:49:17.662667 21563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:49:17.662744 21563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:49:17.672595 21563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46055
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:49:17.673951 21563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:49:17.675293 21563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:49:17.686894 21569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:17.687033 21568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:17.687058 21571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:49:17.688099 21563 server_base.cc:1034] running on GCE node
I20260928 08:49:17.688958 21563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:49:17.690300 21563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:49:17.691510 21563 hybrid_clock.cc:648] HybridClock initialized: now 1790585357691475 us; error 30 us; skew 500 ppm
I20260928 08:49:17.691864 21563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:49:17 dist-test-slave-n6xs krb5kdc[21550](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:49:17.699777 21563 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:49:17.701261 21563 webserver.cc:460] Webserver started at http://127.0.0.1:39069/ using document root <none> and password file <none>
I20260928 08:49:17.701930 21563 fs_manager.cc:359] Metadata directory not provided
I20260928 08:49:17.702057 21563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:49:17.702363 21563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:49:17.705428 21563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "c9b053c93b4a435e99ea91aa26866028"
format_stamp: "Formatted at 2026-09-28 08:49:17 on dist-test-slave-n6xs"
server_key: "0388b6200f77a3f36a4d5c7ab41547b2"
server_key_iv: "0ca984b771017bcd92472f0b9cc3438f"
server_key_version: "encryptionkey@0"
I20260928 08:49:17.706202 21563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "c9b053c93b4a435e99ea91aa26866028"
format_stamp: "Formatted at 2026-09-28 08:49:17 on dist-test-slave-n6xs"
server_key: "0388b6200f77a3f36a4d5c7ab41547b2"
server_key_iv: "0ca984b771017bcd92472f0b9cc3438f"
server_key_version: "encryptionkey@0"
I20260928 08:49:17.711237 21563 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 08:49:17.714613 21578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:49:17.716177 21563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:49:17.716372 21563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "c9b053c93b4a435e99ea91aa26866028"
format_stamp: "Formatted at 2026-09-28 08:49:17 on dist-test-slave-n6xs"
server_key: "0388b6200f77a3f36a4d5c7ab41547b2"
server_key_iv: "0ca984b771017bcd92472f0b9cc3438f"
server_key_version: "encryptionkey@0"
I20260928 08:49:17.716586 21563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:49:17.730273 21563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:49:17.734675 21563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:49:17.734974 21563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:49:17.753726 21563 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41941
I20260928 08:49:17.753752 21629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41941 every 8 connection(s)
I20260928 08:49:17.755117 21563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:49:17.758711 21630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:49:17.763971 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 21563
I20260928 08:49:17.764189 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:49:17.764696 388 external_mini_cluster.cc:1428] Setting key 29a29c0a255d89d9406776509e3f6d98
I20260928 08:49:17.770428 21630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9b053c93b4a435e99ea91aa26866028: Bootstrap starting.
I20260928 08:49:17.773486 21630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9b053c93b4a435e99ea91aa26866028: Neither blocks nor log segments found. Creating new log.
I20260928 08:49:17.774489 21630 log.cc:824] T 00000000000000000000000000000000 P c9b053c93b4a435e99ea91aa26866028: Log is configured to *not* fsync() on all Append() calls
I20260928 08:49:17.777982 21630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9b053c93b4a435e99ea91aa26866028: No bootstrap required, opened a new log
I20260928 08:49:17.778549 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:17.766512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:59831 (local address 127.0.0.1:41941)
0928 08:49:17.767339 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:17.767382 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:17.767438 (+ 56us) server_negotiation.cc:407] Connection header received
0928 08:49:17.769383 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:17.769405 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:17.769767 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:17.770129 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:17.770662 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:17.771998 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:17.773413 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:17.774120 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:17.774312 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:17.775750 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:17.775757 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:17.775770 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:49:17.776575 (+ 805us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:17.776667 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:49:17.776699 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:17.776875 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:17.777124 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 08:49:17.777292 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":324,"threads_started":1}
I20260928 08:49:17.782406 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:49:17.785979 21630 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9b053c93b4a435e99ea91aa26866028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b053c93b4a435e99ea91aa26866028" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41941 } }
I20260928 08:49:17.786336 21630 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9b053c93b4a435e99ea91aa26866028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:49:17.786471 21630 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9b053c93b4a435e99ea91aa26866028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9b053c93b4a435e99ea91aa26866028, State: Initialized, Role: FOLLOWER
I20260928 08:49:17.787119 21630 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9b053c93b4a435e99ea91aa26866028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b053c93b4a435e99ea91aa26866028" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41941 } }
I20260928 08:49:17.787408 21630 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9b053c93b4a435e99ea91aa26866028 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:49:17.787569 21630 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9b053c93b4a435e99ea91aa26866028 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:49:17.787765 21630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9b053c93b4a435e99ea91aa26866028 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:49:17.790688 21630 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9b053c93b4a435e99ea91aa26866028 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b053c93b4a435e99ea91aa26866028" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41941 } }
I20260928 08:49:17.791235 21630 leader_election.cc:302] T 00000000000000000000000000000000 P c9b053c93b4a435e99ea91aa26866028 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9b053c93b4a435e99ea91aa26866028; no voters:
I20260928 08:49:17.791913 21630 leader_election.cc:288] T 00000000000000000000000000000000 P c9b053c93b4a435e99ea91aa26866028 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:49:17.792027 21640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9b053c93b4a435e99ea91aa26866028 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:49:17.792656 21640 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9b053c93b4a435e99ea91aa26866028 [term 1 LEADER]: Becoming Leader. State: Replica: c9b053c93b4a435e99ea91aa26866028, State: Running, Role: LEADER
I20260928 08:49:17.793395 21640 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9b053c93b4a435e99ea91aa26866028 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b053c93b4a435e99ea91aa26866028" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41941 } }
I20260928 08:49:17.794057 21633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:17.786664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:36909 (local address 127.0.0.1:41941)
0928 08:49:17.786870 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:17.786876 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:17.787349 (+ 473us) server_negotiation.cc:407] Connection header received
0928 08:49:17.787539 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:17.787544 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:17.787608 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:17.787736 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:17.788430 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:17.789189 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:17.791107 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:17.791608 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:17.791756 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:17.793082 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:17.793093 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:17.793098 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:49:17.793118 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:17.793189 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:49:17.793205 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:17.793315 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:17.793683 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 08:49:17.793735 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 08:49:17.794435 21630 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9b053c93b4a435e99ea91aa26866028 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:49:17.799218 21641 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9b053c93b4a435e99ea91aa26866028 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9b053c93b4a435e99ea91aa26866028" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b053c93b4a435e99ea91aa26866028" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41941 } } }
I20260928 08:49:17.800038 21641 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9b053c93b4a435e99ea91aa26866028 [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:17.800117 21642 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9b053c93b4a435e99ea91aa26866028 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9b053c93b4a435e99ea91aa26866028. Latest consensus state: current_term: 1 leader_uuid: "c9b053c93b4a435e99ea91aa26866028" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9b053c93b4a435e99ea91aa26866028" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41941 } } }
I20260928 08:49:17.800411 21642 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9b053c93b4a435e99ea91aa26866028 [sys.catalog]: This master's current role is: LEADER
W20260928 08:49:17.806196 21656 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c9b053c93b4a435e99ea91aa26866028: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:49:17.806330 21656 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:49:17.806469 21650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:49:17.808001 21650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:49:17.815488 21650 catalog_manager.cc:1347] Generated new cluster ID: 03c731736c3d4ded8a6e39827594cd88
I20260928 08:49:17.815586 21650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:49:17.843223 21650 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:49:17.845052 21650 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:49:17.855257 21650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9b053c93b4a435e99ea91aa26866028: Generated new TSK 0
I20260928 08:49:17.856172 21650 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:49:17.868561 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 21563
W20260928 08:49:17.870132 21625 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.108:59831
W20260928 08:49:17.870873 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 21563: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:49:17.871042 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 21563
W20260928 08:49:17.887897 21548 connection.cc:535] client connection to 127.0.0.1:41941 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41941): Connection reset by peer (error 104)
2026-09-28T08:49:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1880 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:49:17 dist-test-slave-n6xs krb5kdc[21661](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:49:17 dist-test-slave-n6xs krb5kdc[21661](info): set up 2 sockets
Sep 28 08:49:17 dist-test-slave-n6xs krb5kdc[21661](info): commencing operation
krb5kdc: starting...
W20260928 08:49:18.980235 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:49:19 dist-test-slave-n6xs krb5kdc[21661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:49:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:49:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:49:19.238727 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41431
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41431
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:49:19.565972 21674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:49:19.566233 21674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:49:19.566413 21674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:49:19.573781 21674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:49:19.573882 21674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:49:19.574009 21674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:49:19.574117 21674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:49:19.574193 21674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:49:19.583865 21674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41431
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:49:19.585166 21674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:49:19.586529 21674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:49:19.598333 21680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:19.598441 21679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:19.598467 21682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:49:19.599555 21674 server_base.cc:1034] running on GCE node
I20260928 08:49:19.600347 21674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:49:19.601713 21674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:49:19.602958 21674 hybrid_clock.cc:648] HybridClock initialized: now 1790585359602924 us; error 60 us; skew 500 ppm
I20260928 08:49:19.603304 21674 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:49:19 dist-test-slave-n6xs krb5kdc[21661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:49:19.606527 21674 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:49:19.608217 21674 webserver.cc:460] Webserver started at http://127.0.0.1:35575/ using document root <none> and password file <none>
I20260928 08:49:19.608894 21674 fs_manager.cc:359] Metadata directory not provided
I20260928 08:49:19.609040 21674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:49:19.609368 21674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:49:19.612396 21674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "68f48848f1e4435fa779257eaebd47d0"
format_stamp: "Formatted at 2026-09-28 08:49:19 on dist-test-slave-n6xs"
server_key: "33fd5af69247ae9b0e8733af1597f994"
server_key_iv: "40edae4081b135f0d37c44f0ea581eb3"
server_key_version: "encryptionkey@0"
I20260928 08:49:19.613169 21674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "68f48848f1e4435fa779257eaebd47d0"
format_stamp: "Formatted at 2026-09-28 08:49:19 on dist-test-slave-n6xs"
server_key: "33fd5af69247ae9b0e8733af1597f994"
server_key_iv: "40edae4081b135f0d37c44f0ea581eb3"
server_key_version: "encryptionkey@0"
I20260928 08:49:19.618265 21674 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 08:49:19.621769 21689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:49:19.623148 21674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:49:19.623281 21674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "68f48848f1e4435fa779257eaebd47d0"
format_stamp: "Formatted at 2026-09-28 08:49:19 on dist-test-slave-n6xs"
server_key: "33fd5af69247ae9b0e8733af1597f994"
server_key_iv: "40edae4081b135f0d37c44f0ea581eb3"
server_key_version: "encryptionkey@0"
I20260928 08:49:19.623445 21674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:49:19.629135 21674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:49:19.633347 21674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:49:19.633594 21674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:49:19.652406 21674 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41431
I20260928 08:49:19.652460 21740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41431 every 8 connection(s)
I20260928 08:49:19.653810 21674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:49:19.657874 21741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:49:19.662297 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 21674
I20260928 08:49:19.662524 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:49:19.663100 388 external_mini_cluster.cc:1428] Setting key 19d770dcb86d84b124ad19853fbdd3be
I20260928 08:49:19.671007 21744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:19.665006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:33587 (local address 127.0.0.1:41431)
0928 08:49:19.665801 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:19.665812 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:19.665851 (+ 39us) server_negotiation.cc:407] Connection header received
0928 08:49:19.667487 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:19.667509 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:19.667951 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:19.668289 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:19.669400 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:19.669406 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:19.669419 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:49:19.669433 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:19.669490 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:49:19.669509 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:19.669681 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:19.669786 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 08:49:19.670081 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":368,"threads_started":1}
I20260928 08:49:19.672097 21741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68f48848f1e4435fa779257eaebd47d0: Bootstrap starting.
I20260928 08:49:19.674758 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:49:19.675663 21741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68f48848f1e4435fa779257eaebd47d0: Neither blocks nor log segments found. Creating new log.
I20260928 08:49:19.676985 21741 log.cc:824] T 00000000000000000000000000000000 P 68f48848f1e4435fa779257eaebd47d0: Log is configured to *not* fsync() on all Append() calls
I20260928 08:49:19.679677 21741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68f48848f1e4435fa779257eaebd47d0: No bootstrap required, opened a new log
I20260928 08:49:19.683064 21744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:19.680492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:41677 (local address 127.0.0.1:41431)
0928 08:49:19.680647 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:19.680652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:19.681087 (+ 435us) server_negotiation.cc:407] Connection header received
0928 08:49:19.681222 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:19.681228 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:19.681294 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:19.681406 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:19.682376 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:19.682383 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:19.682387 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:49:19.682405 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:19.682456 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:49:19.682475 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:19.682593 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:19.682681 (+ 88us) server_negotiation.cc:299] Negotiation successful
0928 08:49:19.682737 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 08:49:19.688285 21741 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68f48848f1e4435fa779257eaebd47d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f48848f1e4435fa779257eaebd47d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41431 } }
I20260928 08:49:19.688627 21741 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68f48848f1e4435fa779257eaebd47d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:49:19.688697 21741 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68f48848f1e4435fa779257eaebd47d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68f48848f1e4435fa779257eaebd47d0, State: Initialized, Role: FOLLOWER
I20260928 08:49:19.689221 21741 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68f48848f1e4435fa779257eaebd47d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f48848f1e4435fa779257eaebd47d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41431 } }
I20260928 08:49:19.689424 21741 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68f48848f1e4435fa779257eaebd47d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:49:19.689518 21741 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68f48848f1e4435fa779257eaebd47d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:49:19.689626 21741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68f48848f1e4435fa779257eaebd47d0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:49:19.691612 21741 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68f48848f1e4435fa779257eaebd47d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f48848f1e4435fa779257eaebd47d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41431 } }
I20260928 08:49:19.692094 21741 leader_election.cc:302] T 00000000000000000000000000000000 P 68f48848f1e4435fa779257eaebd47d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68f48848f1e4435fa779257eaebd47d0; no voters:
I20260928 08:49:19.692732 21741 leader_election.cc:288] T 00000000000000000000000000000000 P 68f48848f1e4435fa779257eaebd47d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:49:19.692890 21751 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68f48848f1e4435fa779257eaebd47d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:49:19.693691 21751 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68f48848f1e4435fa779257eaebd47d0 [term 1 LEADER]: Becoming Leader. State: Replica: 68f48848f1e4435fa779257eaebd47d0, State: Running, Role: LEADER
I20260928 08:49:19.694290 21751 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68f48848f1e4435fa779257eaebd47d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f48848f1e4435fa779257eaebd47d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41431 } }
I20260928 08:49:19.694918 21741 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68f48848f1e4435fa779257eaebd47d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:49:19.697285 21753 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68f48848f1e4435fa779257eaebd47d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68f48848f1e4435fa779257eaebd47d0. Latest consensus state: current_term: 1 leader_uuid: "68f48848f1e4435fa779257eaebd47d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f48848f1e4435fa779257eaebd47d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41431 } } }
I20260928 08:49:19.697834 21753 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68f48848f1e4435fa779257eaebd47d0 [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:19.697714 21752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68f48848f1e4435fa779257eaebd47d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68f48848f1e4435fa779257eaebd47d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f48848f1e4435fa779257eaebd47d0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41431 } } }
I20260928 08:49:19.698014 21752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68f48848f1e4435fa779257eaebd47d0 [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:19.698750 21758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:49:19.704913 21758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:49:19.714314 21758 catalog_manager.cc:1347] Generated new cluster ID: 9db6fddc409d4e02b8a78bb0d3f5c84b
I20260928 08:49:19.714423 21758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:49:19.738255 21758 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:49:19.738727 21758 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:49:19.751053 21758 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68f48848f1e4435fa779257eaebd47d0: Generated new TSK 0
I20260928 08:49:19.752010 21758 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:49:19.767221 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 21674
W20260928 08:49:19.768307 21736 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.108:33587
W20260928 08:49:19.769061 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 21674: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:49:19.769224 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 21674
2026-09-28T08:49:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1900 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:49:19 dist-test-slave-n6xs krb5kdc[21772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:49:19 dist-test-slave-n6xs krb5kdc[21772](info): set up 2 sockets
Sep 28 08:49:19 dist-test-slave-n6xs krb5kdc[21772](info): commencing operation
krb5kdc: starting...
W20260928 08:49:20.880281 388 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:49:21 dist-test-slave-n6xs krb5kdc[21772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:49:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:49:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:49:21.134768 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40067
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40067
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:49:21.477346 21785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:49:21.477692 21785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:49:21.477870 21785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:49:21.485334 21785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:49:21.485508 21785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:49:21.485625 21785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:49:21.485769 21785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:49:21.485852 21785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:49:21.495637 21785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40067
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:49:21.497016 21785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:49:21.498390 21785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:49:21.511301 21790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:21.511261 21791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:21.511232 21793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:49:21.512892 21785 server_base.cc:1034] running on GCE node
I20260928 08:49:21.513738 21785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:49:21.515252 21785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:49:21.516451 21785 hybrid_clock.cc:648] HybridClock initialized: now 1790585361516408 us; error 45 us; skew 500 ppm
I20260928 08:49:21.516819 21785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:49:21 dist-test-slave-n6xs krb5kdc[21772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:49:21.520094 21785 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:49:21.521775 21785 webserver.cc:460] Webserver started at http://127.0.0.1:34781/ using document root <none> and password file <none>
I20260928 08:49:21.522473 21785 fs_manager.cc:359] Metadata directory not provided
I20260928 08:49:21.522646 21785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:49:21.522953 21785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:49:21.526031 21785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "3531af6c28fa44ab89c2ceb7a0b111ee"
format_stamp: "Formatted at 2026-09-28 08:49:21 on dist-test-slave-n6xs"
server_key: "93f443aa4d9cc44c573464748cc7c30b"
server_key_iv: "940aa86213d9b300a3b6ccbcdf0fc227"
server_key_version: "encryptionkey@0"
I20260928 08:49:21.526798 21785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "3531af6c28fa44ab89c2ceb7a0b111ee"
format_stamp: "Formatted at 2026-09-28 08:49:21 on dist-test-slave-n6xs"
server_key: "93f443aa4d9cc44c573464748cc7c30b"
server_key_iv: "940aa86213d9b300a3b6ccbcdf0fc227"
server_key_version: "encryptionkey@0"
I20260928 08:49:21.532042 21785 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 08:49:21.535501 21800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:49:21.537062 21785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:49:21.537266 21785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "3531af6c28fa44ab89c2ceb7a0b111ee"
format_stamp: "Formatted at 2026-09-28 08:49:21 on dist-test-slave-n6xs"
server_key: "93f443aa4d9cc44c573464748cc7c30b"
server_key_iv: "940aa86213d9b300a3b6ccbcdf0fc227"
server_key_version: "encryptionkey@0"
I20260928 08:49:21.537503 21785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:49:21.555490 21785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:49:21.560357 21785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:49:21.560676 21785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:49:21.581336 21785 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40067
I20260928 08:49:21.581375 21851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40067 every 8 connection(s)
I20260928 08:49:21.582830 21785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:49:21.586815 21852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:49:21.591349 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 21785
I20260928 08:49:21.591559 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:49:21.592161 388 external_mini_cluster.cc:1428] Setting key b9de698067b6ee667d1e4e5ea6ede921
I20260928 08:49:21.600541 21852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3531af6c28fa44ab89c2ceb7a0b111ee: Bootstrap starting.
I20260928 08:49:21.603686 21852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3531af6c28fa44ab89c2ceb7a0b111ee: Neither blocks nor log segments found. Creating new log.
I20260928 08:49:21.604773 21852 log.cc:824] T 00000000000000000000000000000000 P 3531af6c28fa44ab89c2ceb7a0b111ee: Log is configured to *not* fsync() on all Append() calls
Sep 28 08:49:21 dist-test-slave-n6xs krb5kdc[21772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 08:49:21.608247 21852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3531af6c28fa44ab89c2ceb7a0b111ee: No bootstrap required, opened a new log
I20260928 08:49:21.616791 21855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:21.594113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:51223 (local address 127.0.0.1:40067)
0928 08:49:21.594879 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:21.594892 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:21.594978 (+ 86us) server_negotiation.cc:407] Connection header received
0928 08:49:21.596789 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:21.596811 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:21.597268 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:21.597681 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:21.599232 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:21.600825 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:21.602253 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:21.602982 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:21.603181 (+ 199us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:21.608415 (+ 5234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:21.608425 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:21.608441 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:21.608491 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:21.612121 (+ 3630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:21.613098 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:21.613105 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:21.613111 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:21.613312 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:21.613831 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:21.613834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:21.613837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:21.614378 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:21.614638 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:21.615220 (+ 582us) server_negotiation.cc:299] Negotiation successful
0928 08:49:21.615545 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":334,"threads_started":1}
I20260928 08:49:21.617887 21852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3531af6c28fa44ab89c2ceb7a0b111ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3531af6c28fa44ab89c2ceb7a0b111ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40067 } }
I20260928 08:49:21.618307 21852 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3531af6c28fa44ab89c2ceb7a0b111ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:49:21.618425 21852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3531af6c28fa44ab89c2ceb7a0b111ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3531af6c28fa44ab89c2ceb7a0b111ee, State: Initialized, Role: FOLLOWER
I20260928 08:49:21.619071 21852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3531af6c28fa44ab89c2ceb7a0b111ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3531af6c28fa44ab89c2ceb7a0b111ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40067 } }
I20260928 08:49:21.619333 21852 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3531af6c28fa44ab89c2ceb7a0b111ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:49:21.619510 21852 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3531af6c28fa44ab89c2ceb7a0b111ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:49:21.619711 21852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3531af6c28fa44ab89c2ceb7a0b111ee [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:49:21.621260 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:49:21.622524 21852 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3531af6c28fa44ab89c2ceb7a0b111ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3531af6c28fa44ab89c2ceb7a0b111ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40067 } }
I20260928 08:49:21.623045 21852 leader_election.cc:302] T 00000000000000000000000000000000 P 3531af6c28fa44ab89c2ceb7a0b111ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3531af6c28fa44ab89c2ceb7a0b111ee; no voters:
I20260928 08:49:21.623751 21852 leader_election.cc:288] T 00000000000000000000000000000000 P 3531af6c28fa44ab89c2ceb7a0b111ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:49:21.623925 21857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3531af6c28fa44ab89c2ceb7a0b111ee [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:49:21.630283 21852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3531af6c28fa44ab89c2ceb7a0b111ee [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:49:21.630810 21857 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3531af6c28fa44ab89c2ceb7a0b111ee [term 1 LEADER]: Becoming Leader. State: Replica: 3531af6c28fa44ab89c2ceb7a0b111ee, State: Running, Role: LEADER
I20260928 08:49:21.631551 21857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3531af6c28fa44ab89c2ceb7a0b111ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3531af6c28fa44ab89c2ceb7a0b111ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40067 } }
I20260928 08:49:21.637311 21860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3531af6c28fa44ab89c2ceb7a0b111ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3531af6c28fa44ab89c2ceb7a0b111ee. Latest consensus state: current_term: 1 leader_uuid: "3531af6c28fa44ab89c2ceb7a0b111ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3531af6c28fa44ab89c2ceb7a0b111ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40067 } } }
I20260928 08:49:21.637868 21860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3531af6c28fa44ab89c2ceb7a0b111ee [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:21.640005 21861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3531af6c28fa44ab89c2ceb7a0b111ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3531af6c28fa44ab89c2ceb7a0b111ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3531af6c28fa44ab89c2ceb7a0b111ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40067 } } }
I20260928 08:49:21.640594 21861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3531af6c28fa44ab89c2ceb7a0b111ee [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:21.641484 21867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:49:21.645539 21855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:21.626256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:42167 (local address 127.0.0.1:40067)
0928 08:49:21.626514 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:21.626523 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:21.627078 (+ 555us) server_negotiation.cc:407] Connection header received
0928 08:49:21.627292 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:21.627298 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:21.627378 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:21.627503 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:21.629406 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:21.630381 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:21.632061 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:21.632555 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:21.632695 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:21.635414 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:21.635425 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:21.635432 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:21.635488 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:21.639878 (+ 4390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:21.641176 (+ 1298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:21.641186 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:21.641190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:21.641430 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:21.642151 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:21.642158 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:21.642162 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:21.642748 (+ 586us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:21.644355 (+ 1607us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:21.644644 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 08:49:21.644993 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 08:49:21.648510 21867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:49:21.658018 21867 catalog_manager.cc:1347] Generated new cluster ID: 0c16cc6284db4f12901d6902512cc11f
I20260928 08:49:21.658139 21867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:49:21.675529 21867 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:49:21.677484 21867 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:49:21.686583 21867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3531af6c28fa44ab89c2ceb7a0b111ee: Generated new TSK 0
I20260928 08:49:21.687654 21867 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:49:21.699160 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 21785
I20260928 08:49:21.700877 21847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:49:21.790678 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 21785
W20260928 08:49:21.807878 21770 connection.cc:535] client connection to 127.0.0.1:40067 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40067): Connection reset by peer (error 104)
2026-09-28T08:49:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2019 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:49:21 dist-test-slave-n6xs krb5kdc[21884](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:49:21 dist-test-slave-n6xs krb5kdc[21884](info): set up 2 sockets
Sep 28 08:49:21 dist-test-slave-n6xs krb5kdc[21884](info): commencing operation
krb5kdc: starting...
W20260928 08:49:22.996742 388 mini_kdc.cc:121] Time spent starting KDC: real 1.170s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:49:23 dist-test-slave-n6xs krb5kdc[21884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:49:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:49:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:49:23.247418 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39569
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39569
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:49:23.572723 21897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:49:23.573063 21897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:49:23.573236 21897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:49:23.580441 21897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:49:23.580595 21897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:49:23.580706 21897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:49:23.580826 21897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:49:23.580916 21897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:49:23.590763 21897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39569
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:49:23.592064 21897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:49:23.593287 21897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:49:23.604830 21902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:23.604792 21905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:23.604872 21903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:49:23.605948 21897 server_base.cc:1034] running on GCE node
I20260928 08:49:23.606655 21897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:49:23.608009 21897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:49:23.609213 21897 hybrid_clock.cc:648] HybridClock initialized: now 1790585363609176 us; error 32 us; skew 500 ppm
I20260928 08:49:23.609511 21897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:49:23 dist-test-slave-n6xs krb5kdc[21884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:49:23.618579 21897 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:49:23.620208 21897 webserver.cc:460] Webserver started at http://127.0.0.1:35535/ using document root <none> and password file <none>
I20260928 08:49:23.620889 21897 fs_manager.cc:359] Metadata directory not provided
I20260928 08:49:23.621016 21897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:49:23.621330 21897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:49:23.624333 21897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "6e0d5580b8cb45b1a83809ea23ff69e1"
format_stamp: "Formatted at 2026-09-28 08:49:23 on dist-test-slave-n6xs"
server_key: "abed7117b768f817caa92aff2b8c4b12"
server_key_iv: "a92bcd2fe3807a8dfd58f6f7733e3de8"
server_key_version: "encryptionkey@0"
I20260928 08:49:23.625106 21897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "6e0d5580b8cb45b1a83809ea23ff69e1"
format_stamp: "Formatted at 2026-09-28 08:49:23 on dist-test-slave-n6xs"
server_key: "abed7117b768f817caa92aff2b8c4b12"
server_key_iv: "a92bcd2fe3807a8dfd58f6f7733e3de8"
server_key_version: "encryptionkey@0"
I20260928 08:49:23.630224 21897 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:49:23.633617 21912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:49:23.634927 21897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:49:23.635151 21897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "6e0d5580b8cb45b1a83809ea23ff69e1"
format_stamp: "Formatted at 2026-09-28 08:49:23 on dist-test-slave-n6xs"
server_key: "abed7117b768f817caa92aff2b8c4b12"
server_key_iv: "a92bcd2fe3807a8dfd58f6f7733e3de8"
server_key_version: "encryptionkey@0"
I20260928 08:49:23.635331 21897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:49:23.655447 21897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:49:23.659530 21897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:49:23.659842 21897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:49:23.678030 21897 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39569
I20260928 08:49:23.678087 21963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39569 every 8 connection(s)
I20260928 08:49:23.679440 21897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:49:23.683228 21964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:49:23.684088 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 21897
I20260928 08:49:23.684265 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:49:23.684662 388 external_mini_cluster.cc:1428] Setting key 81c75b3d9d42d23de08300d501a66138
I20260928 08:49:23.696560 21964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e0d5580b8cb45b1a83809ea23ff69e1: Bootstrap starting.
I20260928 08:49:23.697950 21967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:23.686329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:48333 (local address 127.0.0.1:39569)
0928 08:49:23.687068 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:23.687079 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:23.687119 (+ 40us) server_negotiation.cc:407] Connection header received
0928 08:49:23.689075 (+ 1956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:23.689098 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:23.689499 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:23.689812 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:23.690413 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:23.691639 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:23.693587 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:23.694317 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:23.694441 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:23.695690 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:23.695698 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:23.695710 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:49:23.696305 (+ 595us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:23.696379 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:49:23.696402 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:23.696515 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:23.696861 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 08:49:23.697051 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":336,"threads_started":1}
I20260928 08:49:23.699766 21964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e0d5580b8cb45b1a83809ea23ff69e1: Neither blocks nor log segments found. Creating new log.
I20260928 08:49:23.700973 21964 log.cc:824] T 00000000000000000000000000000000 P 6e0d5580b8cb45b1a83809ea23ff69e1: Log is configured to *not* fsync() on all Append() calls
I20260928 08:49:23.701571 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:49:23.703539 21964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e0d5580b8cb45b1a83809ea23ff69e1: No bootstrap required, opened a new log
I20260928 08:49:23.712168 21967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:23.705605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:48379 (local address 127.0.0.1:39569)
0928 08:49:23.705796 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:23.705802 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:23.706214 (+ 412us) server_negotiation.cc:407] Connection header received
0928 08:49:23.706346 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:23.706352 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:23.706442 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:23.706546 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:23.707003 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:23.708171 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:23.709482 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:23.710242 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:23.710380 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:23.711374 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:23.711382 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:23.711391 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:49:23.711417 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:23.711475 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:49:23.711496 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:23.711624 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:23.711775 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 08:49:23.711856 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 08:49:23.712700 21964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e0d5580b8cb45b1a83809ea23ff69e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0d5580b8cb45b1a83809ea23ff69e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39569 } }
I20260928 08:49:23.713079 21964 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e0d5580b8cb45b1a83809ea23ff69e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:49:23.713192 21964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e0d5580b8cb45b1a83809ea23ff69e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e0d5580b8cb45b1a83809ea23ff69e1, State: Initialized, Role: FOLLOWER
I20260928 08:49:23.713761 21964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e0d5580b8cb45b1a83809ea23ff69e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0d5580b8cb45b1a83809ea23ff69e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39569 } }
I20260928 08:49:23.713989 21964 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e0d5580b8cb45b1a83809ea23ff69e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:49:23.714130 21964 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e0d5580b8cb45b1a83809ea23ff69e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:49:23.714294 21964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e0d5580b8cb45b1a83809ea23ff69e1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:49:23.716540 21964 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e0d5580b8cb45b1a83809ea23ff69e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0d5580b8cb45b1a83809ea23ff69e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39569 } }
I20260928 08:49:23.717018 21964 leader_election.cc:302] T 00000000000000000000000000000000 P 6e0d5580b8cb45b1a83809ea23ff69e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e0d5580b8cb45b1a83809ea23ff69e1; no voters:
I20260928 08:49:23.717562 21964 leader_election.cc:288] T 00000000000000000000000000000000 P 6e0d5580b8cb45b1a83809ea23ff69e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:49:23.717689 21974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e0d5580b8cb45b1a83809ea23ff69e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:49:23.718379 21974 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e0d5580b8cb45b1a83809ea23ff69e1 [term 1 LEADER]: Becoming Leader. State: Replica: 6e0d5580b8cb45b1a83809ea23ff69e1, State: Running, Role: LEADER
I20260928 08:49:23.718871 21974 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e0d5580b8cb45b1a83809ea23ff69e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0d5580b8cb45b1a83809ea23ff69e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39569 } }
I20260928 08:49:23.719568 21964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e0d5580b8cb45b1a83809ea23ff69e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:49:23.722116 21976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e0d5580b8cb45b1a83809ea23ff69e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e0d5580b8cb45b1a83809ea23ff69e1. Latest consensus state: current_term: 1 leader_uuid: "6e0d5580b8cb45b1a83809ea23ff69e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0d5580b8cb45b1a83809ea23ff69e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39569 } } }
I20260928 08:49:23.722506 21976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e0d5580b8cb45b1a83809ea23ff69e1 [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:23.722599 21975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e0d5580b8cb45b1a83809ea23ff69e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e0d5580b8cb45b1a83809ea23ff69e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e0d5580b8cb45b1a83809ea23ff69e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39569 } } }
I20260928 08:49:23.722896 21975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e0d5580b8cb45b1a83809ea23ff69e1 [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:23.725065 21980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:49:23.729564 21980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:49:23.738323 21980 catalog_manager.cc:1347] Generated new cluster ID: 47f2a800e4ae49ddb449686bebfb52c4
I20260928 08:49:23.738456 21980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:49:23.759016 21980 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:49:23.760795 21980 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:49:23.781158 21980 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e0d5580b8cb45b1a83809ea23ff69e1: Generated new TSK 0
I20260928 08:49:23.782212 21980 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:49:23.792533 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 21897
W20260928 08:49:23.794337 21959 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.108:48333
W20260928 08:49:23.795153 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 21897: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:49:23.795321 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 21897
W20260928 08:49:23.813153 21882 connection.cc:535] client connection to 127.0.0.1:39569 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39569): Connection reset by peer (error 104)
2026-09-28T08:49:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2005 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:49:23 dist-test-slave-n6xs krb5kdc[21995](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:49:23 dist-test-slave-n6xs krb5kdc[21995](info): set up 2 sockets
Sep 28 08:49:23 dist-test-slave-n6xs krb5kdc[21995](info): commencing operation
krb5kdc: starting...
W20260928 08:49:24.896302 388 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:49:25 dist-test-slave-n6xs krb5kdc[21995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:49:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:49:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 08:49:25.142719 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34145
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:49:25.467202 22008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:49:25.467475 22008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:49:25.467576 22008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:49:25.474722 22008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:49:25.474821 22008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:49:25.474869 22008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:49:25.475003 22008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:49:25.475086 22008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:49:25.484838 22008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34145
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585022442112-388-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:49:25.486179 22008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:49:25.487398 22008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:49:25.499053 22013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:49:25.504850 22008 server_base.cc:1034] running on GCE node
W20260928 08:49:25.504935 22014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:25.505077 22016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:49:25.505960 22008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:49:25.507266 22008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:49:25.508481 22008 hybrid_clock.cc:648] HybridClock initialized: now 1790585365508419 us; error 36 us; skew 500 ppm
I20260928 08:49:25.508831 22008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:49:25 dist-test-slave-n6xs krb5kdc[21995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585365, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:49:25.511899 22008 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 08:49:25.513222 22008 webserver.cc:460] Webserver started at http://127.0.0.1:45791/ using document root <none> and password file <none>
I20260928 08:49:25.513906 22008 fs_manager.cc:359] Metadata directory not provided
I20260928 08:49:25.514051 22008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:49:25.514340 22008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:49:25.517302 22008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "276e056b2e5643df9406ee04685b7a99"
format_stamp: "Formatted at 2026-09-28 08:49:25 on dist-test-slave-n6xs"
server_key: "f57d1c43db65dde2760aaead85781f0b"
server_key_iv: "ccbff8ab15b6372cf79f309b11ccf362"
server_key_version: "encryptionkey@0"
I20260928 08:49:25.518044 22008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "276e056b2e5643df9406ee04685b7a99"
format_stamp: "Formatted at 2026-09-28 08:49:25 on dist-test-slave-n6xs"
server_key: "f57d1c43db65dde2760aaead85781f0b"
server_key_iv: "ccbff8ab15b6372cf79f309b11ccf362"
server_key_version: "encryptionkey@0"
I20260928 08:49:25.523245 22008 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 08:49:25.526669 22023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:49:25.528059 22008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:49:25.528235 22008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "276e056b2e5643df9406ee04685b7a99"
format_stamp: "Formatted at 2026-09-28 08:49:25 on dist-test-slave-n6xs"
server_key: "f57d1c43db65dde2760aaead85781f0b"
server_key_iv: "ccbff8ab15b6372cf79f309b11ccf362"
server_key_version: "encryptionkey@0"
I20260928 08:49:25.528424 22008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:49:25.534075 22008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:49:25.537958 22008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:49:25.538239 22008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:49:25.558241 22008 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34145
I20260928 08:49:25.558293 22074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34145 every 8 connection(s)
I20260928 08:49:25.559679 22008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:49:25.564337 22075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:49:25.566674 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 22008
I20260928 08:49:25.566854 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:49:25.567385 388 external_mini_cluster.cc:1428] Setting key df573669f14ff7c85c208487af523521
I20260928 08:49:25.576210 22078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:25.569289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:54457 (local address 127.0.0.1:34145)
0928 08:49:25.570101 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:25.570113 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:25.570260 (+ 147us) server_negotiation.cc:407] Connection header received
0928 08:49:25.572625 (+ 2365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:25.572647 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:25.573009 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:25.573427 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:25.574517 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:25.574524 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:25.574537 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:49:25.574551 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:25.574610 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:49:25.574635 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:25.574874 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:25.575022 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 08:49:25.575205 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":766,"threads_started":1}
I20260928 08:49:25.580403 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:49:25.580641 22075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 276e056b2e5643df9406ee04685b7a99: Bootstrap starting.
I20260928 08:49:25.583892 22075 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 276e056b2e5643df9406ee04685b7a99: Neither blocks nor log segments found. Creating new log.
I20260928 08:49:25.585183 22075 log.cc:824] T 00000000000000000000000000000000 P 276e056b2e5643df9406ee04685b7a99: Log is configured to *not* fsync() on all Append() calls
I20260928 08:49:25.587416 22078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:25.584866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.108:50221 (local address 127.0.0.1:34145)
0928 08:49:25.585070 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:25.585075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:25.585446 (+ 371us) server_negotiation.cc:407] Connection header received
0928 08:49:25.585701 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:25.585705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:25.585749 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:25.585867 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:25.586791 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:25.586796 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:25.586801 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:49:25.586824 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:25.586860 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:49:25.586875 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:25.586995 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:25.587089 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 08:49:25.587148 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 08:49:25.588426 22075 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 276e056b2e5643df9406ee04685b7a99: No bootstrap required, opened a new log
I20260928 08:49:25.595672 22075 raft_consensus.cc:348] T 00000000000000000000000000000000 P 276e056b2e5643df9406ee04685b7a99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276e056b2e5643df9406ee04685b7a99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34145 } }
I20260928 08:49:25.596161 22075 raft_consensus.cc:374] T 00000000000000000000000000000000 P 276e056b2e5643df9406ee04685b7a99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:49:25.596278 22075 raft_consensus.cc:729] T 00000000000000000000000000000000 P 276e056b2e5643df9406ee04685b7a99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 276e056b2e5643df9406ee04685b7a99, State: Initialized, Role: FOLLOWER
I20260928 08:49:25.596903 22075 consensus_queue.cc:260] T 00000000000000000000000000000000 P 276e056b2e5643df9406ee04685b7a99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276e056b2e5643df9406ee04685b7a99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34145 } }
I20260928 08:49:25.597164 22075 raft_consensus.cc:388] T 00000000000000000000000000000000 P 276e056b2e5643df9406ee04685b7a99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:49:25.597334 22075 raft_consensus.cc:482] T 00000000000000000000000000000000 P 276e056b2e5643df9406ee04685b7a99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:49:25.597512 22075 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 276e056b2e5643df9406ee04685b7a99 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:49:25.599896 22075 raft_consensus.cc:504] T 00000000000000000000000000000000 P 276e056b2e5643df9406ee04685b7a99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276e056b2e5643df9406ee04685b7a99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34145 } }
I20260928 08:49:25.600473 22075 leader_election.cc:302] T 00000000000000000000000000000000 P 276e056b2e5643df9406ee04685b7a99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 276e056b2e5643df9406ee04685b7a99; no voters:
I20260928 08:49:25.601082 22075 leader_election.cc:288] T 00000000000000000000000000000000 P 276e056b2e5643df9406ee04685b7a99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:49:25.601233 22085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 276e056b2e5643df9406ee04685b7a99 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:49:25.601943 22085 raft_consensus.cc:686] T 00000000000000000000000000000000 P 276e056b2e5643df9406ee04685b7a99 [term 1 LEADER]: Becoming Leader. State: Replica: 276e056b2e5643df9406ee04685b7a99, State: Running, Role: LEADER
I20260928 08:49:25.602504 22085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 276e056b2e5643df9406ee04685b7a99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276e056b2e5643df9406ee04685b7a99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34145 } }
I20260928 08:49:25.603377 22075 sys_catalog.cc:564] T 00000000000000000000000000000000 P 276e056b2e5643df9406ee04685b7a99 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:49:25.605710 22087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 276e056b2e5643df9406ee04685b7a99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 276e056b2e5643df9406ee04685b7a99. Latest consensus state: current_term: 1 leader_uuid: "276e056b2e5643df9406ee04685b7a99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276e056b2e5643df9406ee04685b7a99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34145 } } }
I20260928 08:49:25.606215 22087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 276e056b2e5643df9406ee04685b7a99 [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:25.606828 22086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 276e056b2e5643df9406ee04685b7a99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "276e056b2e5643df9406ee04685b7a99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "276e056b2e5643df9406ee04685b7a99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34145 } } }
I20260928 08:49:25.607213 22086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 276e056b2e5643df9406ee04685b7a99 [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:25.608397 22092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:49:25.613400 22092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:49:25.623544 22092 catalog_manager.cc:1347] Generated new cluster ID: b0dd06c93c8841e683633cfe7b33d0d9
I20260928 08:49:25.623677 22092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:49:25.653515 22092 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:49:25.653920 22092 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:49:25.659950 22092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 276e056b2e5643df9406ee04685b7a99: Generated new TSK 0
I20260928 08:49:25.660884 22092 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:49:25.671169 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 22008
W20260928 08:49:25.672271 22070 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.108:54457
W20260928 08:49:25.673058 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 22008: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:49:25.673215 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 22008
2026-09-28T08:49:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1877 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:49:25 dist-test-slave-n6xs krb5kdc[22106](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:49:25 dist-test-slave-n6xs krb5kdc[22106](info): set up 2 sockets
Sep 28 08:49:25 dist-test-slave-n6xs krb5kdc[22106](info): commencing operation
krb5kdc: starting...
W20260928 08:49:26.772195 388 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:49:26 dist-test-slave-n6xs krb5kdc[22106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:49:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:49:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:49:27.008790 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44149
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:44425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:44149
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:49:27.331878 22119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:49:27.332199 22119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:49:27.332425 22119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:49:27.339574 22119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:49:27.339671 22119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:49:27.339772 22119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:49:27.339915 22119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:49:27.339977 22119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:49:27.349792 22119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:44425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:44149
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:44149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:49:27.351020 22119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:49:27.352365 22119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:49:27.363670 22124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:27.363700 22127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:27.363646 22125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:49:27.364775 22119 server_base.cc:1034] running on GCE node
I20260928 08:49:27.365469 22119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:49:27.366767 22119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:49:27.367959 22119 hybrid_clock.cc:648] HybridClock initialized: now 1790585367367910 us; error 43 us; skew 500 ppm
I20260928 08:49:27.368274 22119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:49:27 dist-test-slave-n6xs krb5kdc[22106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585367, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:49:27.371335 22119 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:49:27.372745 22119 webserver.cc:460] Webserver started at http://127.0.97.62:33185/ using document root <none> and password file <none>
I20260928 08:49:27.373387 22119 fs_manager.cc:359] Metadata directory not provided
I20260928 08:49:27.373523 22119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:49:27.373821 22119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:49:27.376652 22119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "dcdc640234004424aa0635b4759f689d"
format_stamp: "Formatted at 2026-09-28 08:49:27 on dist-test-slave-n6xs"
server_key: "a84930617eb9efa3f86ae4b6025cd801"
server_key_iv: "1edd11e71e922a626fe0312ddf9d667c"
server_key_version: "encryptionkey@0"
I20260928 08:49:27.377377 22119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "dcdc640234004424aa0635b4759f689d"
format_stamp: "Formatted at 2026-09-28 08:49:27 on dist-test-slave-n6xs"
server_key: "a84930617eb9efa3f86ae4b6025cd801"
server_key_iv: "1edd11e71e922a626fe0312ddf9d667c"
server_key_version: "encryptionkey@0"
I20260928 08:49:27.382164 22119 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260928 08:49:27.385540 22134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:49:27.386766 22119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 08:49:27.386950 22119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "dcdc640234004424aa0635b4759f689d"
format_stamp: "Formatted at 2026-09-28 08:49:27 on dist-test-slave-n6xs"
server_key: "a84930617eb9efa3f86ae4b6025cd801"
server_key_iv: "1edd11e71e922a626fe0312ddf9d667c"
server_key_version: "encryptionkey@0"
I20260928 08:49:27.387141 22119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:49:27.409716 22119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:49:27.414464 22119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:49:27.414764 22119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:49:27.439406 22119 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:44149
I20260928 08:49:27.439512 22185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:44149 every 8 connection(s)
I20260928 08:49:27.440822 22119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:49:27.442137 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 22119
I20260928 08:49:27.442317 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:49:27.442874 388 external_mini_cluster.cc:1428] Setting key 82631a4b5493c589d240ce9c2876f22b
I20260928 08:49:27.445304 22186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 08:49:27 dist-test-slave-n6xs krb5kdc[22106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:49:27.458235 22186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcdc640234004424aa0635b4759f689d: Bootstrap starting.
I20260928 08:49:27.461550 22186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcdc640234004424aa0635b4759f689d: Neither blocks nor log segments found. Creating new log.
I20260928 08:49:27.462541 22186 log.cc:824] T 00000000000000000000000000000000 P dcdc640234004424aa0635b4759f689d: Log is configured to *not* fsync() on all Append() calls
I20260928 08:49:27.465368 22186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcdc640234004424aa0635b4759f689d: No bootstrap required, opened a new log
I20260928 08:49:27.466310 22189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:27.444580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60169 (local address 127.0.97.62:44149)
0928 08:49:27.445337 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:27.445348 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:27.445416 (+ 68us) server_negotiation.cc:407] Connection header received
0928 08:49:27.447053 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:27.447072 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:27.447426 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:27.447747 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:27.449532 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:27.450867 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:27.452684 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:27.453418 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:27.453537 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:27.458523 (+ 4986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:27.458532 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:27.458544 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:27.458581 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:27.461850 (+ 3269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:27.462696 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:27.462707 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:27.462715 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:27.462890 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:27.463456 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:27.463460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:27.463462 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:27.464080 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:27.464420 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:27.464991 (+ 571us) server_negotiation.cc:299] Negotiation successful
0928 08:49:27.465299 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":365,"threads_started":1}
I20260928 08:49:27.470289 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:49:27.473249 22186 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcdc640234004424aa0635b4759f689d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcdc640234004424aa0635b4759f689d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44149 } }
I20260928 08:49:27.473579 22186 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcdc640234004424aa0635b4759f689d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:49:27.473702 22186 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcdc640234004424aa0635b4759f689d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcdc640234004424aa0635b4759f689d, State: Initialized, Role: FOLLOWER
I20260928 08:49:27.474234 22186 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcdc640234004424aa0635b4759f689d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcdc640234004424aa0635b4759f689d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44149 } }
I20260928 08:49:27.474473 22186 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcdc640234004424aa0635b4759f689d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:49:27.474637 22186 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcdc640234004424aa0635b4759f689d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:49:27.474880 22186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcdc640234004424aa0635b4759f689d [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:49:27.478010 22186 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcdc640234004424aa0635b4759f689d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcdc640234004424aa0635b4759f689d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44149 } }
I20260928 08:49:27.478591 22186 leader_election.cc:302] T 00000000000000000000000000000000 P dcdc640234004424aa0635b4759f689d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcdc640234004424aa0635b4759f689d; no voters:
I20260928 08:49:27.479235 22186 leader_election.cc:288] T 00000000000000000000000000000000 P dcdc640234004424aa0635b4759f689d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:49:27.479295 22196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcdc640234004424aa0635b4759f689d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:49:27.481045 22186 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcdc640234004424aa0635b4759f689d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:49:27.481674 22196 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcdc640234004424aa0635b4759f689d [term 1 LEADER]: Becoming Leader. State: Replica: dcdc640234004424aa0635b4759f689d, State: Running, Role: LEADER
I20260928 08:49:27.482332 22196 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcdc640234004424aa0635b4759f689d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcdc640234004424aa0635b4759f689d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44149 } }
I20260928 08:49:27.485137 22198 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcdc640234004424aa0635b4759f689d [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcdc640234004424aa0635b4759f689d. Latest consensus state: current_term: 1 leader_uuid: "dcdc640234004424aa0635b4759f689d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcdc640234004424aa0635b4759f689d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44149 } } }
I20260928 08:49:27.485495 22198 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcdc640234004424aa0635b4759f689d [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:27.486145 22197 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcdc640234004424aa0635b4759f689d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcdc640234004424aa0635b4759f689d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcdc640234004424aa0635b4759f689d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 44149 } } }
I20260928 08:49:27.486419 22197 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcdc640234004424aa0635b4759f689d [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:27.488669 22204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:49:27.494323 22189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:27.474673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46451 (local address 127.0.97.62:44149)
0928 08:49:27.474885 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:27.474893 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:27.475398 (+ 505us) server_negotiation.cc:407] Connection header received
0928 08:49:27.475568 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:27.475572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:27.475625 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:27.475710 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:27.477468 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:27.478296 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:27.480266 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:27.481172 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:27.481325 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:27.484857 (+ 3532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:27.484868 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:27.484875 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:27.484930 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:27.488981 (+ 4051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:27.490068 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:27.490076 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:27.490079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:27.490260 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:27.491142 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:27.491153 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:27.491160 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:27.491676 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:27.493238 (+ 1562us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:27.493472 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 08:49:27.493734 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":56}
I20260928 08:49:27.494876 22204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:49:27.504760 22204 catalog_manager.cc:1347] Generated new cluster ID: 14f45932f82146d094b1410b322346c0
I20260928 08:49:27.504877 22204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:49:27.527647 22204 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:49:27.529260 22204 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:49:27.536268 22204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcdc640234004424aa0635b4759f689d: Generated new TSK 0
I20260928 08:49:27.537114 22204 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:49:27.551338 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 22119
I20260928 08:49:27.552671 22181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:49:27.644960 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 22119
W20260928 08:49:27.662732 22104 connection.cc:535] client connection to 127.0.97.62:44149 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:44149): Connection reset by peer (error 104)
2026-09-28T08:49:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1971 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:49:27 dist-test-slave-n6xs krb5kdc[22218](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:49:27 dist-test-slave-n6xs krb5kdc[22218](info): set up 2 sockets
Sep 28 08:49:27 dist-test-slave-n6xs krb5kdc[22218](info): commencing operation
krb5kdc: starting...
W20260928 08:49:28.780628 388 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:49:28 dist-test-slave-n6xs krb5kdc[22218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585368, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:49:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:49:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:49:29.044137 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42877
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:32907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:42877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:49:29.384148 22231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:49:29.384521 22231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:49:29.384704 22231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:49:29.392536 22231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:49:29.392729 22231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:49:29.392853 22231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:49:29.392984 22231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:49:29.393075 22231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:49:29.403365 22231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:32907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:42877
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:42877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:49:29.404825 22231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:49:29.406247 22231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:49:29.419284 22237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:29.419611 22239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:29.419832 22236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:49:29.420677 22231 server_base.cc:1034] running on GCE node
I20260928 08:49:29.421674 22231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:49:29.423593 22231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:49:29.424808 22231 hybrid_clock.cc:648] HybridClock initialized: now 1790585369424691 us; error 103 us; skew 500 ppm
I20260928 08:49:29.425197 22231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:49:29 dist-test-slave-n6xs krb5kdc[22218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585369, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:49:29.428550 22231 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:49:29.430011 22231 webserver.cc:460] Webserver started at http://127.0.97.62:36659/ using document root <none> and password file <none>
I20260928 08:49:29.430737 22231 fs_manager.cc:359] Metadata directory not provided
I20260928 08:49:29.430882 22231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:49:29.431186 22231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:49:29.434201 22231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "625a90952c1b48e88ea287c7519fc52b"
format_stamp: "Formatted at 2026-09-28 08:49:29 on dist-test-slave-n6xs"
server_key: "f2d910fe06103f17b0d02178f4720a57"
server_key_iv: "69780cc5c3cfd5de900edc6d0ebd8f21"
server_key_version: "encryptionkey@0"
I20260928 08:49:29.434959 22231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "625a90952c1b48e88ea287c7519fc52b"
format_stamp: "Formatted at 2026-09-28 08:49:29 on dist-test-slave-n6xs"
server_key: "f2d910fe06103f17b0d02178f4720a57"
server_key_iv: "69780cc5c3cfd5de900edc6d0ebd8f21"
server_key_version: "encryptionkey@0"
I20260928 08:49:29.440151 22231 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 08:49:29.443464 22246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:49:29.444847 22231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:49:29.445037 22231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "625a90952c1b48e88ea287c7519fc52b"
format_stamp: "Formatted at 2026-09-28 08:49:29 on dist-test-slave-n6xs"
server_key: "f2d910fe06103f17b0d02178f4720a57"
server_key_iv: "69780cc5c3cfd5de900edc6d0ebd8f21"
server_key_version: "encryptionkey@0"
I20260928 08:49:29.445243 22231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:49:29.474880 22231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:49:29.479348 22231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:49:29.479640 22231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:49:29.497988 22231 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:42877
I20260928 08:49:29.498093 22297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:42877 every 8 connection(s)
I20260928 08:49:29.499365 22231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:49:29.503139 22298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:49:29.509133 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 22231
I20260928 08:49:29.509325 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:49:29.509737 388 external_mini_cluster.cc:1428] Setting key d8f33ad42c3a153d9afa0b52de58207d
I20260928 08:49:29.514124 22298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 625a90952c1b48e88ea287c7519fc52b: Bootstrap starting.
I20260928 08:49:29.517972 22298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 625a90952c1b48e88ea287c7519fc52b: Neither blocks nor log segments found. Creating new log.
I20260928 08:49:29.518949 22298 log.cc:824] T 00000000000000000000000000000000 P 625a90952c1b48e88ea287c7519fc52b: Log is configured to *not* fsync() on all Append() calls
I20260928 08:49:29.521909 22298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 625a90952c1b48e88ea287c7519fc52b: No bootstrap required, opened a new log
I20260928 08:49:29.524410 22301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:29.511617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41601 (local address 127.0.97.62:42877)
0928 08:49:29.512446 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:29.512461 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:29.512506 (+ 45us) server_negotiation.cc:407] Connection header received
0928 08:49:29.514698 (+ 2192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:29.514724 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:29.515169 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:29.515604 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:29.516170 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:29.518097 (+ 1927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:29.519447 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:29.520334 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:29.520513 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:29.521731 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:29.521741 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:29.521758 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:49:29.522453 (+ 695us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:29.522559 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:49:29.522590 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:29.522745 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:29.523006 (+ 261us) server_negotiation.cc:299] Negotiation successful
0928 08:49:29.523162 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":394,"threads_started":1}
I20260928 08:49:29.528543 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:49:29.530885 22298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 625a90952c1b48e88ea287c7519fc52b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625a90952c1b48e88ea287c7519fc52b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42877 } }
I20260928 08:49:29.531260 22298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 625a90952c1b48e88ea287c7519fc52b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:49:29.531373 22298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 625a90952c1b48e88ea287c7519fc52b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 625a90952c1b48e88ea287c7519fc52b, State: Initialized, Role: FOLLOWER
I20260928 08:49:29.532133 22298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 625a90952c1b48e88ea287c7519fc52b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625a90952c1b48e88ea287c7519fc52b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42877 } }
I20260928 08:49:29.532406 22298 raft_consensus.cc:388] T 00000000000000000000000000000000 P 625a90952c1b48e88ea287c7519fc52b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:49:29.532567 22298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 625a90952c1b48e88ea287c7519fc52b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:49:29.532740 22298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 625a90952c1b48e88ea287c7519fc52b [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:49:29.535674 22298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 625a90952c1b48e88ea287c7519fc52b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625a90952c1b48e88ea287c7519fc52b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42877 } }
I20260928 08:49:29.536227 22298 leader_election.cc:302] T 00000000000000000000000000000000 P 625a90952c1b48e88ea287c7519fc52b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 625a90952c1b48e88ea287c7519fc52b; no voters:
I20260928 08:49:29.536852 22298 leader_election.cc:288] T 00000000000000000000000000000000 P 625a90952c1b48e88ea287c7519fc52b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:49:29.536957 22308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 625a90952c1b48e88ea287c7519fc52b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:49:29.537628 22308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 625a90952c1b48e88ea287c7519fc52b [term 1 LEADER]: Becoming Leader. State: Replica: 625a90952c1b48e88ea287c7519fc52b, State: Running, Role: LEADER
I20260928 08:49:29.538282 22308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 625a90952c1b48e88ea287c7519fc52b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625a90952c1b48e88ea287c7519fc52b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42877 } }
I20260928 08:49:29.540231 22308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 625a90952c1b48e88ea287c7519fc52b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 625a90952c1b48e88ea287c7519fc52b. Latest consensus state: current_term: 1 leader_uuid: "625a90952c1b48e88ea287c7519fc52b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625a90952c1b48e88ea287c7519fc52b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42877 } } }
I20260928 08:49:29.540512 22308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 625a90952c1b48e88ea287c7519fc52b [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:29.541416 22310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 625a90952c1b48e88ea287c7519fc52b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "625a90952c1b48e88ea287c7519fc52b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "625a90952c1b48e88ea287c7519fc52b" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 42877 } } }
I20260928 08:49:29.541924 22301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:29.533124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54129 (local address 127.0.97.62:42877)
0928 08:49:29.533346 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:29.533352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:29.533697 (+ 345us) server_negotiation.cc:407] Connection header received
0928 08:49:29.533871 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:29.533874 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:29.533930 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:29.534023 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:29.534600 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:29.535351 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:29.537331 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:29.538109 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:29.538258 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:29.539974 (+ 1716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:29.539981 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:29.539986 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:49:29.540005 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:29.540076 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:49:29.540103 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:29.540218 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:29.540766 (+ 548us) server_negotiation.cc:299] Negotiation successful
0928 08:49:29.540818 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 08:49:29.542023 22310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 625a90952c1b48e88ea287c7519fc52b [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:29.545153 22298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 625a90952c1b48e88ea287c7519fc52b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:49:29.545341 22312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:49:29.552191 22312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:49:29.562521 22312 catalog_manager.cc:1347] Generated new cluster ID: 990670be9703422b95eca0bc728086a7
I20260928 08:49:29.562652 22312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:49:29.579591 22312 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:49:29.581406 22312 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:49:29.592912 22312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 625a90952c1b48e88ea287c7519fc52b: Generated new TSK 0
I20260928 08:49:29.593941 22312 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:49:29.601658 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 22231
W20260928 08:49:29.602965 22293 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41601
W20260928 08:49:29.603855 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 22231: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:49:29.604001 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 22231
W20260928 08:49:29.622427 22216 connection.cc:535] client connection to 127.0.97.62:42877 recv error: Network error: failed to read from TLS socket (remote: 127.0.97.62:42877): Connection reset by peer (error 104)
2026-09-28T08:49:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1959 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:49:29 dist-test-slave-n6xs krb5kdc[22329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:49:29 dist-test-slave-n6xs krb5kdc[22329](info): set up 2 sockets
Sep 28 08:49:29 dist-test-slave-n6xs krb5kdc[22329](info): commencing operation
krb5kdc: starting...
W20260928 08:49:30.720343 388 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.006s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:49:30 dist-test-slave-n6xs krb5kdc[22329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585370, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:49:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:49:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:49:30.978539 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34125
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:49:31.323117 22342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:49:31.323413 22342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:49:31.323596 22342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:49:31.331125 22342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:49:31.331257 22342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:49:31.331391 22342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:49:31.331542 22342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:49:31.331616 22342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:49:31.342860 22342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34125
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:49:31.344393 22342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:49:31.345844 22342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:49:31.358278 22348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:31.358309 22350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:31.358503 22347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:49:31.359232 22342 server_base.cc:1034] running on GCE node
I20260928 08:49:31.360148 22342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:49:31.361589 22342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:49:31.362838 22342 hybrid_clock.cc:648] HybridClock initialized: now 1790585371362791 us; error 60 us; skew 500 ppm
I20260928 08:49:31.363220 22342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:49:31 dist-test-slave-n6xs krb5kdc[22329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585371, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:49:31.366516 22342 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:49:31.368234 22342 webserver.cc:460] Webserver started at http://127.0.97.62:41723/ using document root <none> and password file <none>
I20260928 08:49:31.368971 22342 fs_manager.cc:359] Metadata directory not provided
I20260928 08:49:31.369134 22342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:49:31.369449 22342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:49:31.372599 22342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "793bdd28a6f44a91919928aa493bd038"
format_stamp: "Formatted at 2026-09-28 08:49:31 on dist-test-slave-n6xs"
server_key: "ef7dedd6b69ee3e8f819063d56d7806a"
server_key_iv: "15832560f753bf3138bf48b964479a7c"
server_key_version: "encryptionkey@0"
I20260928 08:49:31.373380 22342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "793bdd28a6f44a91919928aa493bd038"
format_stamp: "Formatted at 2026-09-28 08:49:31 on dist-test-slave-n6xs"
server_key: "ef7dedd6b69ee3e8f819063d56d7806a"
server_key_iv: "15832560f753bf3138bf48b964479a7c"
server_key_version: "encryptionkey@0"
I20260928 08:49:31.379065 22342 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 08:49:31.382704 22357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:49:31.384171 22342 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:49:31.384347 22342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "793bdd28a6f44a91919928aa493bd038"
format_stamp: "Formatted at 2026-09-28 08:49:31 on dist-test-slave-n6xs"
server_key: "ef7dedd6b69ee3e8f819063d56d7806a"
server_key_iv: "15832560f753bf3138bf48b964479a7c"
server_key_version: "encryptionkey@0"
I20260928 08:49:31.384562 22342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:49:31.390300 22342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:49:31.394320 22342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:49:31.394610 22342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:49:31.413905 22342 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34125
I20260928 08:49:31.413973 22408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34125 every 8 connection(s)
I20260928 08:49:31.416069 22342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:49:31.420490 22409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:49:31.423866 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 22342
I20260928 08:49:31.424039 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:49:31.424458 388 external_mini_cluster.cc:1428] Setting key c557c7fc9cb4c9c2d2332c177cfdaa40
I20260928 08:49:31.432224 22412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:31.426236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60125 (local address 127.0.97.62:34125)
0928 08:49:31.426980 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:31.426992 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:31.427074 (+ 82us) server_negotiation.cc:407] Connection header received
0928 08:49:31.428744 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:31.428766 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:31.429114 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:31.429501 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:31.430536 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:31.430541 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:31.430553 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:49:31.430567 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:31.430620 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:49:31.430637 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:31.430857 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:31.430984 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 08:49:31.431293 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":331,"threads_started":1}
I20260928 08:49:31.432956 22409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 793bdd28a6f44a91919928aa493bd038: Bootstrap starting.
I20260928 08:49:31.435833 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:49:31.436416 22409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 793bdd28a6f44a91919928aa493bd038: Neither blocks nor log segments found. Creating new log.
I20260928 08:49:31.437448 22409 log.cc:824] T 00000000000000000000000000000000 P 793bdd28a6f44a91919928aa493bd038: Log is configured to *not* fsync() on all Append() calls
I20260928 08:49:31.440563 22409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 793bdd28a6f44a91919928aa493bd038: No bootstrap required, opened a new log
I20260928 08:49:31.442835 22412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:31.440105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43263 (local address 127.0.97.62:34125)
0928 08:49:31.440253 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:31.440258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:31.440807 (+ 549us) server_negotiation.cc:407] Connection header received
0928 08:49:31.441023 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:31.441031 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:31.441089 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:31.441193 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:31.442175 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:31.442180 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:31.442185 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:49:31.442203 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:31.442247 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:49:31.442261 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:31.442390 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:31.442483 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 08:49:31.442568 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 08:49:31.449275 22409 raft_consensus.cc:348] T 00000000000000000000000000000000 P 793bdd28a6f44a91919928aa493bd038 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793bdd28a6f44a91919928aa493bd038" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34125 } }
I20260928 08:49:31.449720 22409 raft_consensus.cc:374] T 00000000000000000000000000000000 P 793bdd28a6f44a91919928aa493bd038 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:49:31.449829 22409 raft_consensus.cc:729] T 00000000000000000000000000000000 P 793bdd28a6f44a91919928aa493bd038 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793bdd28a6f44a91919928aa493bd038, State: Initialized, Role: FOLLOWER
I20260928 08:49:31.450443 22409 consensus_queue.cc:260] T 00000000000000000000000000000000 P 793bdd28a6f44a91919928aa493bd038 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793bdd28a6f44a91919928aa493bd038" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34125 } }
I20260928 08:49:31.450700 22409 raft_consensus.cc:388] T 00000000000000000000000000000000 P 793bdd28a6f44a91919928aa493bd038 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:49:31.450850 22409 raft_consensus.cc:482] T 00000000000000000000000000000000 P 793bdd28a6f44a91919928aa493bd038 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:49:31.451043 22409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 793bdd28a6f44a91919928aa493bd038 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:49:31.453722 22409 raft_consensus.cc:504] T 00000000000000000000000000000000 P 793bdd28a6f44a91919928aa493bd038 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793bdd28a6f44a91919928aa493bd038" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34125 } }
I20260928 08:49:31.454349 22409 leader_election.cc:302] T 00000000000000000000000000000000 P 793bdd28a6f44a91919928aa493bd038 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 793bdd28a6f44a91919928aa493bd038; no voters:
I20260928 08:49:31.455021 22409 leader_election.cc:288] T 00000000000000000000000000000000 P 793bdd28a6f44a91919928aa493bd038 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:49:31.455227 22419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 793bdd28a6f44a91919928aa493bd038 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:49:31.455946 22419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 793bdd28a6f44a91919928aa493bd038 [term 1 LEADER]: Becoming Leader. State: Replica: 793bdd28a6f44a91919928aa493bd038, State: Running, Role: LEADER
I20260928 08:49:31.456501 22419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 793bdd28a6f44a91919928aa493bd038 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793bdd28a6f44a91919928aa493bd038" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34125 } }
I20260928 08:49:31.457161 22409 sys_catalog.cc:564] T 00000000000000000000000000000000 P 793bdd28a6f44a91919928aa493bd038 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:49:31.459796 22420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 793bdd28a6f44a91919928aa493bd038 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "793bdd28a6f44a91919928aa493bd038" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793bdd28a6f44a91919928aa493bd038" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34125 } } }
I20260928 08:49:31.460065 22421 sys_catalog.cc:455] T 00000000000000000000000000000000 P 793bdd28a6f44a91919928aa493bd038 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 793bdd28a6f44a91919928aa493bd038. Latest consensus state: current_term: 1 leader_uuid: "793bdd28a6f44a91919928aa493bd038" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793bdd28a6f44a91919928aa493bd038" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34125 } } }
I20260928 08:49:31.460332 22420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 793bdd28a6f44a91919928aa493bd038 [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:31.460362 22421 sys_catalog.cc:458] T 00000000000000000000000000000000 P 793bdd28a6f44a91919928aa493bd038 [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:31.463485 22428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:49:31.468760 22428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:49:31.477851 22428 catalog_manager.cc:1347] Generated new cluster ID: 1589ae413a3b4f929396a4b2abe26af2
I20260928 08:49:31.478052 22428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:49:31.498862 22428 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:49:31.499368 22428 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:49:31.508455 22428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 793bdd28a6f44a91919928aa493bd038: Generated new TSK 0
I20260928 08:49:31.509344 22428 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:49:31.517879 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 22342
W20260928 08:49:31.519362 22404 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60125
W20260928 08:49:31.520072 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 22342: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:49:31.520248 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 22342
2026-09-28T08:49:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1914 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:49:31 dist-test-slave-n6xs krb5kdc[22440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:49:31 dist-test-slave-n6xs krb5kdc[22440](info): set up 2 sockets
Sep 28 08:49:31 dist-test-slave-n6xs krb5kdc[22440](info): commencing operation
krb5kdc: starting...
W20260928 08:49:32.624187 388 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.006s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:49:32 dist-test-slave-n6xs krb5kdc[22440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:49:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:49:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:49:32.871011 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34667
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:34667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:49:33.193840 22453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:49:33.194165 22453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:49:33.194329 22453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:49:33.201478 22453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:49:33.201615 22453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:49:33.201711 22453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:49:33.201809 22453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:49:33.201891 22453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:49:33.211577 22453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:34667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:34667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:49:33.212841 22453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:49:33.214175 22453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:49:33.225641 22458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:33.225600 22459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:33.225558 22461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:49:33.226779 22453 server_base.cc:1034] running on GCE node
I20260928 08:49:33.227530 22453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:49:33.228965 22453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:49:33.230208 22453 hybrid_clock.cc:648] HybridClock initialized: now 1790585373230177 us; error 35 us; skew 500 ppm
I20260928 08:49:33.230561 22453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:49:33 dist-test-slave-n6xs krb5kdc[22440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585373, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:49:33.233958 22453 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:49:33.235571 22453 webserver.cc:460] Webserver started at http://127.0.97.62:44503/ using document root <none> and password file <none>
I20260928 08:49:33.236347 22453 fs_manager.cc:359] Metadata directory not provided
I20260928 08:49:33.236507 22453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:49:33.236850 22453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:49:33.239919 22453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "e53d11d9943b4feea7129b9775e1ed0d"
format_stamp: "Formatted at 2026-09-28 08:49:33 on dist-test-slave-n6xs"
server_key: "8a174731c9d42838f3ff1f2fbdfc09be"
server_key_iv: "08b09a4e9e9d11fc545005261ab16a53"
server_key_version: "encryptionkey@0"
I20260928 08:49:33.240707 22453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "e53d11d9943b4feea7129b9775e1ed0d"
format_stamp: "Formatted at 2026-09-28 08:49:33 on dist-test-slave-n6xs"
server_key: "8a174731c9d42838f3ff1f2fbdfc09be"
server_key_iv: "08b09a4e9e9d11fc545005261ab16a53"
server_key_version: "encryptionkey@0"
I20260928 08:49:33.246104 22453 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:49:33.249755 22468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:49:33.251013 22453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:49:33.251211 22453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "e53d11d9943b4feea7129b9775e1ed0d"
format_stamp: "Formatted at 2026-09-28 08:49:33 on dist-test-slave-n6xs"
server_key: "8a174731c9d42838f3ff1f2fbdfc09be"
server_key_iv: "08b09a4e9e9d11fc545005261ab16a53"
server_key_version: "encryptionkey@0"
I20260928 08:49:33.251405 22453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:49:33.268592 22453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:49:33.272616 22453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:49:33.272886 22453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:49:33.290856 22453 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:34667
I20260928 08:49:33.290889 22519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:34667 every 8 connection(s)
I20260928 08:49:33.292166 22453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:49:33.294611 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 22453
I20260928 08:49:33.294791 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:49:33.295332 388 external_mini_cluster.cc:1428] Setting key a03d6d1be3fe0212d9d5350597d62394
I20260928 08:49:33.296231 22520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:49:33.308001 22520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e53d11d9943b4feea7129b9775e1ed0d: Bootstrap starting.
Sep 28 08:49:33 dist-test-slave-n6xs krb5kdc[22440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585372, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:49:33.311054 22520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e53d11d9943b4feea7129b9775e1ed0d: Neither blocks nor log segments found. Creating new log.
I20260928 08:49:33.312135 22520 log.cc:824] T 00000000000000000000000000000000 P e53d11d9943b4feea7129b9775e1ed0d: Log is configured to *not* fsync() on all Append() calls
I20260928 08:49:33.314744 22520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e53d11d9943b4feea7129b9775e1ed0d: No bootstrap required, opened a new log
I20260928 08:49:33.316711 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:33.297090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52463 (local address 127.0.97.62:34667)
0928 08:49:33.297833 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:33.297844 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:33.297882 (+ 38us) server_negotiation.cc:407] Connection header received
0928 08:49:33.299502 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:33.299523 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:33.299962 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:33.300400 (+ 438us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:33.301834 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:33.303242 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:33.304641 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:33.305335 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:33.305558 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:33.309707 (+ 4149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:33.309714 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:33.309727 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:33.309772 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:33.312974 (+ 3202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:33.313655 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:33.313661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:33.313667 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:33.313823 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:33.314191 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:33.314197 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:33.314200 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:33.314659 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:33.314948 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:33.315373 (+ 425us) server_negotiation.cc:299] Negotiation successful
0928 08:49:33.315749 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":341,"threads_started":1}
I20260928 08:49:33.320621 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:49:33.323122 22520 raft_consensus.cc:348] T 00000000000000000000000000000000 P e53d11d9943b4feea7129b9775e1ed0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53d11d9943b4feea7129b9775e1ed0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34667 } }
I20260928 08:49:33.323534 22520 raft_consensus.cc:374] T 00000000000000000000000000000000 P e53d11d9943b4feea7129b9775e1ed0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:49:33.323652 22520 raft_consensus.cc:729] T 00000000000000000000000000000000 P e53d11d9943b4feea7129b9775e1ed0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e53d11d9943b4feea7129b9775e1ed0d, State: Initialized, Role: FOLLOWER
I20260928 08:49:33.324259 22520 consensus_queue.cc:260] T 00000000000000000000000000000000 P e53d11d9943b4feea7129b9775e1ed0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53d11d9943b4feea7129b9775e1ed0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34667 } }
I20260928 08:49:33.324497 22520 raft_consensus.cc:388] T 00000000000000000000000000000000 P e53d11d9943b4feea7129b9775e1ed0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:49:33.324658 22520 raft_consensus.cc:482] T 00000000000000000000000000000000 P e53d11d9943b4feea7129b9775e1ed0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:49:33.324860 22520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e53d11d9943b4feea7129b9775e1ed0d [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:49:33.328416 22520 raft_consensus.cc:504] T 00000000000000000000000000000000 P e53d11d9943b4feea7129b9775e1ed0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53d11d9943b4feea7129b9775e1ed0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34667 } }
I20260928 08:49:33.328967 22520 leader_election.cc:302] T 00000000000000000000000000000000 P e53d11d9943b4feea7129b9775e1ed0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e53d11d9943b4feea7129b9775e1ed0d; no voters:
I20260928 08:49:33.329607 22520 leader_election.cc:288] T 00000000000000000000000000000000 P e53d11d9943b4feea7129b9775e1ed0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:49:33.329674 22530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e53d11d9943b4feea7129b9775e1ed0d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:49:33.331303 22530 raft_consensus.cc:686] T 00000000000000000000000000000000 P e53d11d9943b4feea7129b9775e1ed0d [term 1 LEADER]: Becoming Leader. State: Replica: e53d11d9943b4feea7129b9775e1ed0d, State: Running, Role: LEADER
I20260928 08:49:33.331727 22520 sys_catalog.cc:564] T 00000000000000000000000000000000 P e53d11d9943b4feea7129b9775e1ed0d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:49:33.332005 22530 consensus_queue.cc:237] T 00000000000000000000000000000000 P e53d11d9943b4feea7129b9775e1ed0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53d11d9943b4feea7129b9775e1ed0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34667 } }
I20260928 08:49:33.336298 22531 sys_catalog.cc:455] T 00000000000000000000000000000000 P e53d11d9943b4feea7129b9775e1ed0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e53d11d9943b4feea7129b9775e1ed0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53d11d9943b4feea7129b9775e1ed0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34667 } } }
I20260928 08:49:33.336691 22531 sys_catalog.cc:458] T 00000000000000000000000000000000 P e53d11d9943b4feea7129b9775e1ed0d [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:33.337188 22532 sys_catalog.cc:455] T 00000000000000000000000000000000 P e53d11d9943b4feea7129b9775e1ed0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e53d11d9943b4feea7129b9775e1ed0d. Latest consensus state: current_term: 1 leader_uuid: "e53d11d9943b4feea7129b9775e1ed0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e53d11d9943b4feea7129b9775e1ed0d" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 34667 } } }
I20260928 08:49:33.337481 22532 sys_catalog.cc:458] T 00000000000000000000000000000000 P e53d11d9943b4feea7129b9775e1ed0d [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:33.346046 22537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:49:33.346081 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:33.324867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37283 (local address 127.0.97.62:34667)
0928 08:49:33.325068 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:33.325076 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:33.325583 (+ 507us) server_negotiation.cc:407] Connection header received
0928 08:49:33.325750 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:33.325754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:33.325818 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:33.325896 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:33.327645 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:33.328738 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:33.331677 (+ 2939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:33.332455 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:33.332584 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:33.336273 (+ 3689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:33.336287 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:33.336295 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:33.336346 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:33.341084 (+ 4738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:33.342238 (+ 1154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:33.342246 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:33.342252 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:33.342468 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:33.343001 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:33.343009 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:33.343014 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:33.343486 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:33.343789 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:33.345163 (+ 1374us) server_negotiation.cc:299] Negotiation successful
0928 08:49:33.345483 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 08:49:33.350987 22537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:49:33.360224 22537 catalog_manager.cc:1347] Generated new cluster ID: 36679a675f70407996373dddcb9d7c69
I20260928 08:49:33.360358 22537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:49:33.372938 22537 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:49:33.374195 22537 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:49:33.386973 22537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e53d11d9943b4feea7129b9775e1ed0d: Generated new TSK 0
I20260928 08:49:33.388031 22537 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:49:33.395375 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 22453
I20260928 08:49:33.396409 22515 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:49:33.489413 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 22453
2026-09-28T08:49:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1971 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:49:33 dist-test-slave-n6xs krb5kdc[22552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:49:33 dist-test-slave-n6xs krb5kdc[22552](info): set up 2 sockets
Sep 28 08:49:33 dist-test-slave-n6xs krb5kdc[22552](info): commencing operation
krb5kdc: starting...
W20260928 08:49:34.612346 388 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:49:34 dist-test-slave-n6xs krb5kdc[22552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:49:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:49:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:49:34.869282 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43243
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:41849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:49:35.201031 22565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:49:35.201310 22565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:49:35.201400 22565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:49:35.208914 22565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:49:35.209040 22565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:49:35.209086 22565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:49:35.209146 22565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:49:35.209192 22565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:49:35.219113 22565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:41849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:49:35.220464 22565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:49:35.221710 22565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:49:35.234234 22570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:35.234308 22573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:35.234274 22571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:49:35.235165 22565 server_base.cc:1034] running on GCE node
I20260928 08:49:35.236065 22565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:49:35.237489 22565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:49:35.238736 22565 hybrid_clock.cc:648] HybridClock initialized: now 1790585375238694 us; error 34 us; skew 500 ppm
I20260928 08:49:35.239109 22565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:49:35 dist-test-slave-n6xs krb5kdc[22552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585375, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:49:35.246951 22565 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:49:35.248770 22565 webserver.cc:460] Webserver started at http://127.0.97.62:35419/ using document root <none> and password file <none>
I20260928 08:49:35.249500 22565 fs_manager.cc:359] Metadata directory not provided
I20260928 08:49:35.249655 22565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:49:35.249971 22565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:49:35.253098 22565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "89b59907c303487da275cffc06cd606c"
format_stamp: "Formatted at 2026-09-28 08:49:35 on dist-test-slave-n6xs"
server_key: "a384d5f6e299d7e8f2a4c53b9bd7315d"
server_key_iv: "1f26a2e547804a80eabc7ce25d6800c3"
server_key_version: "encryptionkey@0"
I20260928 08:49:35.253854 22565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "89b59907c303487da275cffc06cd606c"
format_stamp: "Formatted at 2026-09-28 08:49:35 on dist-test-slave-n6xs"
server_key: "a384d5f6e299d7e8f2a4c53b9bd7315d"
server_key_iv: "1f26a2e547804a80eabc7ce25d6800c3"
server_key_version: "encryptionkey@0"
I20260928 08:49:35.259003 22565 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260928 08:49:35.262892 22580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:49:35.264689 22565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:49:35.264828 22565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "89b59907c303487da275cffc06cd606c"
format_stamp: "Formatted at 2026-09-28 08:49:35 on dist-test-slave-n6xs"
server_key: "a384d5f6e299d7e8f2a4c53b9bd7315d"
server_key_iv: "1f26a2e547804a80eabc7ce25d6800c3"
server_key_version: "encryptionkey@0"
I20260928 08:49:35.265005 22565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:49:35.275946 22565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:49:35.280260 22565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:49:35.280484 22565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:49:35.299582 22565 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43243
I20260928 08:49:35.299584 22631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43243 every 8 connection(s)
I20260928 08:49:35.301069 22565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:49:35.304081 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 22565
I20260928 08:49:35.304265 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:49:35.304669 388 external_mini_cluster.cc:1428] Setting key 89aeffdcc8b3fdc2d88eef11b1fd1b77
I20260928 08:49:35.305190 22632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:49:35.318024 22632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89b59907c303487da275cffc06cd606c: Bootstrap starting.
I20260928 08:49:35.318629 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:35.306560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48089 (local address 127.0.97.62:43243)
0928 08:49:35.307631 (+ 1071us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:35.307642 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:35.307682 (+ 40us) server_negotiation.cc:407] Connection header received
0928 08:49:35.310128 (+ 2446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:35.310150 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:35.310570 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:35.310989 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:35.311547 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:35.312941 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:35.314265 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:35.314944 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:35.315163 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:35.316375 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:35.316384 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:35.316396 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:49:35.316928 (+ 532us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:35.317001 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:49:35.317025 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:35.317171 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:35.317380 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 08:49:35.317639 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":859,"thread_start_us":435,"threads_started":1}
I20260928 08:49:35.321332 22632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89b59907c303487da275cffc06cd606c: Neither blocks nor log segments found. Creating new log.
I20260928 08:49:35.322476 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:49:35.322532 22632 log.cc:824] T 00000000000000000000000000000000 P 89b59907c303487da275cffc06cd606c: Log is configured to *not* fsync() on all Append() calls
I20260928 08:49:35.325349 22632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89b59907c303487da275cffc06cd606c: No bootstrap required, opened a new log
I20260928 08:49:35.333783 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:35.326876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36525 (local address 127.0.97.62:43243)
0928 08:49:35.327093 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:35.327099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:35.327596 (+ 497us) server_negotiation.cc:407] Connection header received
0928 08:49:35.327798 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:35.327804 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:35.327908 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:35.328032 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:35.328551 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:35.329665 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:35.330989 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:35.331713 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:35.331876 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:35.332970 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:35.332977 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:35.332982 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:49:35.333028 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:35.333079 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:49:35.333106 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:35.333209 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:35.333333 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 08:49:35.333437 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 08:49:35.334830 22632 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89b59907c303487da275cffc06cd606c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b59907c303487da275cffc06cd606c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43243 } }
I20260928 08:49:35.335270 22632 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89b59907c303487da275cffc06cd606c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:49:35.335386 22632 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89b59907c303487da275cffc06cd606c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89b59907c303487da275cffc06cd606c, State: Initialized, Role: FOLLOWER
I20260928 08:49:35.336014 22632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89b59907c303487da275cffc06cd606c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b59907c303487da275cffc06cd606c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43243 } }
I20260928 08:49:35.336231 22632 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89b59907c303487da275cffc06cd606c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:49:35.336337 22632 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89b59907c303487da275cffc06cd606c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:49:35.336500 22632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89b59907c303487da275cffc06cd606c [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:49:35.338999 22632 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89b59907c303487da275cffc06cd606c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b59907c303487da275cffc06cd606c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43243 } }
I20260928 08:49:35.339444 22632 leader_election.cc:302] T 00000000000000000000000000000000 P 89b59907c303487da275cffc06cd606c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89b59907c303487da275cffc06cd606c; no voters:
I20260928 08:49:35.340143 22632 leader_election.cc:288] T 00000000000000000000000000000000 P 89b59907c303487da275cffc06cd606c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:49:35.340269 22642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89b59907c303487da275cffc06cd606c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:49:35.341822 22642 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89b59907c303487da275cffc06cd606c [term 1 LEADER]: Becoming Leader. State: Replica: 89b59907c303487da275cffc06cd606c, State: Running, Role: LEADER
I20260928 08:49:35.342361 22632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89b59907c303487da275cffc06cd606c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:49:35.342376 22642 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89b59907c303487da275cffc06cd606c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b59907c303487da275cffc06cd606c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43243 } }
I20260928 08:49:35.345453 22644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89b59907c303487da275cffc06cd606c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89b59907c303487da275cffc06cd606c. Latest consensus state: current_term: 1 leader_uuid: "89b59907c303487da275cffc06cd606c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b59907c303487da275cffc06cd606c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43243 } } }
I20260928 08:49:35.345809 22644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89b59907c303487da275cffc06cd606c [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:35.346906 22643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89b59907c303487da275cffc06cd606c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89b59907c303487da275cffc06cd606c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89b59907c303487da275cffc06cd606c" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43243 } } }
I20260928 08:49:35.347242 22643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89b59907c303487da275cffc06cd606c [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:35.352636 22652 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:49:35.358182 22652 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:49:35.367205 22652 catalog_manager.cc:1347] Generated new cluster ID: 6b0e604acc85477eadf966bd874af59b
I20260928 08:49:35.367369 22652 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:49:35.409091 22652 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:49:35.411055 22652 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:49:35.422514 22652 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89b59907c303487da275cffc06cd606c: Generated new TSK 0
I20260928 08:49:35.423606 22652 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:49:35.439388 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 22565
W20260928 08:49:35.441059 22627 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48089
W20260928 08:49:35.441787 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 22565: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:49:35.441922 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 22565
2026-09-28T08:49:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1946 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:49:35 dist-test-slave-n6xs krb5kdc[22663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:49:35 dist-test-slave-n6xs krb5kdc[22663](info): set up 2 sockets
Sep 28 08:49:35 dist-test-slave-n6xs krb5kdc[22663](info): commencing operation
krb5kdc: starting...
W20260928 08:49:36.552510 388 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.007s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:49:36 dist-test-slave-n6xs krb5kdc[22663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585376, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:49:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:49:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:49:36.831562 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43189
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:36103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:43189
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:49:37.185209 22676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:49:37.185637 22676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:49:37.185846 22676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:49:37.193629 22676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:49:37.193852 22676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:49:37.193984 22676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:49:37.194115 22676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:49:37.194211 22676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:49:37.206748 22676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:36103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:43189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:43189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:49:37.208413 22676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:49:37.209888 22676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:49:37.223639 22681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:37.223559 22684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:37.223595 22682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:49:37.224888 22676 server_base.cc:1034] running on GCE node
I20260928 08:49:37.225869 22676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:49:37.227511 22676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:49:37.228787 22676 hybrid_clock.cc:648] HybridClock initialized: now 1790585377228653 us; error 124 us; skew 500 ppm
I20260928 08:49:37.229192 22676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:49:37 dist-test-slave-n6xs krb5kdc[22663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585377, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:49:37.232656 22676 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:49:37.234326 22676 webserver.cc:460] Webserver started at http://127.0.97.62:46615/ using document root <none> and password file <none>
I20260928 08:49:37.235100 22676 fs_manager.cc:359] Metadata directory not provided
I20260928 08:49:37.235209 22676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:49:37.235494 22676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:49:37.239087 22676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "37f488f5a3564c1e83e6d7021b1ba1ed"
format_stamp: "Formatted at 2026-09-28 08:49:37 on dist-test-slave-n6xs"
server_key: "f53c155d90f039baf7a97006f88eebfe"
server_key_iv: "61d65d33550ea4d72a423056d13d5cdd"
server_key_version: "encryptionkey@0"
I20260928 08:49:37.240029 22676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "37f488f5a3564c1e83e6d7021b1ba1ed"
format_stamp: "Formatted at 2026-09-28 08:49:37 on dist-test-slave-n6xs"
server_key: "f53c155d90f039baf7a97006f88eebfe"
server_key_iv: "61d65d33550ea4d72a423056d13d5cdd"
server_key_version: "encryptionkey@0"
I20260928 08:49:37.246358 22676 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 08:49:37.250399 22691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:49:37.252177 22676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:49:37.252404 22676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "37f488f5a3564c1e83e6d7021b1ba1ed"
format_stamp: "Formatted at 2026-09-28 08:49:37 on dist-test-slave-n6xs"
server_key: "f53c155d90f039baf7a97006f88eebfe"
server_key_iv: "61d65d33550ea4d72a423056d13d5cdd"
server_key_version: "encryptionkey@0"
I20260928 08:49:37.252631 22676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:49:37.258947 22676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:49:37.263979 22676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:49:37.264339 22676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:49:37.284448 22676 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:43189
I20260928 08:49:37.284493 22742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:43189 every 8 connection(s)
I20260928 08:49:37.286023 22676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:49:37.288966 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 22676
I20260928 08:49:37.289158 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:49:37.289628 388 external_mini_cluster.cc:1428] Setting key df163f77bada1390dd835a2cd2a4c1d4
I20260928 08:49:37.290246 22743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:49:37.300089 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:37.291579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45509 (local address 127.0.97.62:43189)
0928 08:49:37.292485 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:37.292516 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:37.292569 (+ 53us) server_negotiation.cc:407] Connection header received
0928 08:49:37.295791 (+ 3222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:37.295850 (+ 59us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:37.296389 (+ 539us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:37.296927 (+ 538us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:37.297996 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:37.298010 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:37.298033 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:49:37.298055 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:37.298132 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:49:37.298171 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:37.298408 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:37.298547 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 08:49:37.298770 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":435,"threads_started":1}
I20260928 08:49:37.303975 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:49:37.305637 22743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37f488f5a3564c1e83e6d7021b1ba1ed: Bootstrap starting.
I20260928 08:49:37.309099 22743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37f488f5a3564c1e83e6d7021b1ba1ed: Neither blocks nor log segments found. Creating new log.
I20260928 08:49:37.310266 22743 log.cc:824] T 00000000000000000000000000000000 P 37f488f5a3564c1e83e6d7021b1ba1ed: Log is configured to *not* fsync() on all Append() calls
I20260928 08:49:37.311200 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:37.308682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47355 (local address 127.0.97.62:43189)
0928 08:49:37.308874 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:37.308881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:37.309191 (+ 310us) server_negotiation.cc:407] Connection header received
0928 08:49:37.309396 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:37.309404 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:37.309470 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:37.309594 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:37.310537 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:37.310543 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:37.310547 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 08:49:37.310568 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:37.310614 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 08:49:37.310630 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:37.310751 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:37.310845 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 08:49:37.310911 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 08:49:37.313576 22743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37f488f5a3564c1e83e6d7021b1ba1ed: No bootstrap required, opened a new log
I20260928 08:49:37.321275 22743 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37f488f5a3564c1e83e6d7021b1ba1ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f488f5a3564c1e83e6d7021b1ba1ed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43189 } }
I20260928 08:49:37.321707 22743 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37f488f5a3564c1e83e6d7021b1ba1ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 08:49:37.321868 22743 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37f488f5a3564c1e83e6d7021b1ba1ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37f488f5a3564c1e83e6d7021b1ba1ed, State: Initialized, Role: FOLLOWER
I20260928 08:49:37.322536 22743 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37f488f5a3564c1e83e6d7021b1ba1ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f488f5a3564c1e83e6d7021b1ba1ed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43189 } }
I20260928 08:49:37.322786 22743 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37f488f5a3564c1e83e6d7021b1ba1ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 08:49:37.322994 22743 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37f488f5a3564c1e83e6d7021b1ba1ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 08:49:37.323180 22743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37f488f5a3564c1e83e6d7021b1ba1ed [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:49:37.325559 22743 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37f488f5a3564c1e83e6d7021b1ba1ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f488f5a3564c1e83e6d7021b1ba1ed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43189 } }
I20260928 08:49:37.326239 22743 leader_election.cc:302] T 00000000000000000000000000000000 P 37f488f5a3564c1e83e6d7021b1ba1ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37f488f5a3564c1e83e6d7021b1ba1ed; no voters:
I20260928 08:49:37.326864 22743 leader_election.cc:288] T 00000000000000000000000000000000 P 37f488f5a3564c1e83e6d7021b1ba1ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 08:49:37.327019 22753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37f488f5a3564c1e83e6d7021b1ba1ed [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:49:37.327771 22753 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37f488f5a3564c1e83e6d7021b1ba1ed [term 1 LEADER]: Becoming Leader. State: Replica: 37f488f5a3564c1e83e6d7021b1ba1ed, State: Running, Role: LEADER
I20260928 08:49:37.328529 22753 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37f488f5a3564c1e83e6d7021b1ba1ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f488f5a3564c1e83e6d7021b1ba1ed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43189 } }
I20260928 08:49:37.329406 22743 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37f488f5a3564c1e83e6d7021b1ba1ed [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:49:37.331409 22754 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37f488f5a3564c1e83e6d7021b1ba1ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37f488f5a3564c1e83e6d7021b1ba1ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f488f5a3564c1e83e6d7021b1ba1ed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43189 } } }
I20260928 08:49:37.331930 22754 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37f488f5a3564c1e83e6d7021b1ba1ed [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:37.332237 22755 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37f488f5a3564c1e83e6d7021b1ba1ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37f488f5a3564c1e83e6d7021b1ba1ed. Latest consensus state: current_term: 1 leader_uuid: "37f488f5a3564c1e83e6d7021b1ba1ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37f488f5a3564c1e83e6d7021b1ba1ed" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 43189 } } }
I20260928 08:49:37.332559 22755 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37f488f5a3564c1e83e6d7021b1ba1ed [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:37.334487 22758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:49:37.340080 22758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:49:37.350291 22758 catalog_manager.cc:1347] Generated new cluster ID: 5b7983a12e4349379e00ac7f8560a5ec
I20260928 08:49:37.350433 22758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:49:37.367331 22758 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:49:37.367866 22758 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:49:37.380019 22758 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37f488f5a3564c1e83e6d7021b1ba1ed: Generated new TSK 0
I20260928 08:49:37.380983 22758 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:49:37.388976 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 22676
W20260928 08:49:37.390141 22738 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45509
W20260928 08:49:37.390949 388 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 22676: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 08:49:37.391144 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 22676
2026-09-28T08:49:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1952 ms)
[----------] 18 tests from AuthTokenIssuingTest (35218 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:49:37 dist-test-slave-n6xs krb5kdc[22774](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:49:37 dist-test-slave-n6xs krb5kdc[22774](info): set up 2 sockets
Sep 28 08:49:37 dist-test-slave-n6xs krb5kdc[22774](info): commencing operation
krb5kdc: starting...
W20260928 08:49:38.504339 388 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.001s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:49:38 dist-test-slave-n6xs krb5kdc[22774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:49:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:49:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:49:38.779165 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40259
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40259,127.0.97.61:34399,127.0.97.60:46481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:49:39.134694 22787 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 08:49:39.135032 22787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:49:39.135097 22787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:49:39.135185 22787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:49:39.142513 22787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:49:39.142620 22787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:49:39.142679 22787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:49:39.142724 22787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:49:39.142787 22787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:49:39.152750 22787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40259,127.0.97.61:34399,127.0.97.60:46481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:40259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:49:39.154110 22787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:49:39.155427 22787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:49:39.167860 22793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:39.168155 22795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:39.167834 22792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:49:39.168772 22787 server_base.cc:1034] running on GCE node
I20260928 08:49:39.169667 22787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:49:39.171159 22787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:49:39.172420 22787 hybrid_clock.cc:648] HybridClock initialized: now 1790585379172363 us; error 57 us; skew 500 ppm
I20260928 08:49:39.172775 22787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:49:39 dist-test-slave-n6xs krb5kdc[22774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:49:39.175992 22787 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:49:39.177613 22787 webserver.cc:460] Webserver started at http://127.0.97.62:45195/ using document root <none> and password file <none>
I20260928 08:49:39.178383 22787 fs_manager.cc:359] Metadata directory not provided
I20260928 08:49:39.178522 22787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:49:39.178858 22787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:49:39.181977 22787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "8366fd63aacf45bdb7bcb6f9aa55c00f"
format_stamp: "Formatted at 2026-09-28 08:49:39 on dist-test-slave-n6xs"
server_key: "710eeca65668365b51ea66573f2feeee"
server_key_iv: "f036927ac3a4749f15fa9dcebac77934"
server_key_version: "encryptionkey@0"
I20260928 08:49:39.182802 22787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "8366fd63aacf45bdb7bcb6f9aa55c00f"
format_stamp: "Formatted at 2026-09-28 08:49:39 on dist-test-slave-n6xs"
server_key: "710eeca65668365b51ea66573f2feeee"
server_key_iv: "f036927ac3a4749f15fa9dcebac77934"
server_key_version: "encryptionkey@0"
I20260928 08:49:39.188266 22787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 08:49:39.191695 22802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:49:39.193166 22787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:49:39.193354 22787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "8366fd63aacf45bdb7bcb6f9aa55c00f"
format_stamp: "Formatted at 2026-09-28 08:49:39 on dist-test-slave-n6xs"
server_key: "710eeca65668365b51ea66573f2feeee"
server_key_iv: "f036927ac3a4749f15fa9dcebac77934"
server_key_version: "encryptionkey@0"
I20260928 08:49:39.193598 22787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:49:39.208608 22787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:49:39.213501 22787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:49:39.213828 22787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:49:39.233280 22787 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:40259
I20260928 08:49:39.233299 22853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:40259 every 8 connection(s)
I20260928 08:49:39.234727 22787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:49:39.235782 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 22787
I20260928 08:49:39.236027 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:49:39.236617 388 external_mini_cluster.cc:1428] Setting key 5b24c68c7c421c717bc04c7d1505c4c4
I20260928 08:49:39.238756 22854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:49:39.245502 22854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40259 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 28 08:49:39 dist-test-slave-n6xs krb5kdc[22774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:49:39.273466 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:39.248845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40259 (local address 127.0.0.1:50822)
0928 08:49:39.249718 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:39.251007 (+ 1289us) client_negotiation.cc:174] Beginning negotiation
0928 08:49:39.252945 (+ 1938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:49:39.253971 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:49:39.253976 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:49:39.254614 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:49:39.254970 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:39.254990 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:39.257501 (+ 2511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:39.257505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:39.258632 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:39.258645 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:39.260069 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:39.260074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:39.260224 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:39.260902 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:49:39.260925 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:49:39.264582 (+ 3657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:49:39.269357 (+ 4775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:39.269368 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:39.269379 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:39.269784 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:39.270276 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:39.270279 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:39.270281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:39.270444 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:39.271457 (+ 1013us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:49:39.271463 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:49:39.271666 (+ 203us) client_negotiation.cc:769] Sending connection context
0928 08:49:39.271947 (+ 281us) client_negotiation.cc:240] Negotiation successful
0928 08:49:39.272383 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":374,"threads_started":1}
I20260928 08:49:39.273506 22858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:39.250283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50822 (local address 127.0.97.62:40259)
0928 08:49:39.251210 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:39.251221 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:39.251277 (+ 56us) server_negotiation.cc:407] Connection header received
0928 08:49:39.253363 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:39.253373 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:39.253716 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:39.253940 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:39.255222 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:39.257325 (+ 2103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:39.258796 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:39.259836 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:39.260031 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:39.264890 (+ 4859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:39.264904 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:39.264926 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:39.264974 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:39.269170 (+ 4196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:39.269930 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:39.269942 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:39.269948 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:39.270152 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:39.270566 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:39.270574 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:39.270580 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:39.271317 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:39.271522 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:39.272122 (+ 600us) server_negotiation.cc:299] Negotiation successful
0928 08:49:39.272605 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":137,"server-negotiator.queue_time_us":701,"thread_start_us":482,"threads_started":1}
I20260928 08:49:39.277068 22854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34399 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:49:39.278290 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:39.277855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34399 (local address 127.0.0.1:45752)
0928 08:49:39.278041 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:39.278159 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 08:49:39.278595 22804 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34399: connect: Connection refused (error 111)
I20260928 08:49:39.280320 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:39.279962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34399 (local address 127.0.0.1:45758)
0928 08:49:39.280108 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:39.280196 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 08:49:39.280540 22854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34399: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34399: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260928 08:49:39.299510 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:34399
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40259,127.0.97.61:34399,127.0.97.60:46481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:49:39.329851 22854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34399 } attempt: 1
I20260928 08:49:39.331094 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:39.330659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34399 (local address 127.0.0.1:45770)
0928 08:49:39.330856 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:39.330947 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 08:49:39.332192 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:39.331799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34399 (local address 127.0.0.1:45772)
0928 08:49:39.331982 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:39.332059 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 08:49:39.332420 22854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34399: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34399: connect: Connection refused (error 111)
I20260928 08:49:39.400735 22854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34399 } attempt: 2
I20260928 08:49:39.401942 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:39.401446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34399 (local address 127.0.0.1:45788)
0928 08:49:39.401713 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:39.401799 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 08:49:39.403082 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:39.402715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34399 (local address 127.0.0.1:45798)
0928 08:49:39.402859 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:39.402938 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 08:49:39.403337 22854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34399: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34399: connect: Connection refused (error 111)
I20260928 08:49:39.494663 22854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34399 } attempt: 3
I20260928 08:49:39.495949 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:39.495407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34399 (local address 127.0.0.1:45810)
0928 08:49:39.495635 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:39.495738 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 08:49:39.497259 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:39.496750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34399 (local address 127.0.0.1:45826)
0928 08:49:39.496901 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:39.496992 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 08:49:39.497522 22854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34399: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34399: connect: Connection refused (error 111)
I20260928 08:49:39.640821 22854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34399 } attempt: 4
I20260928 08:49:39.642005 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:39.641473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34399 (local address 127.0.0.1:45842)
0928 08:49:39.641772 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:39.641878 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260928 08:49:39.643220 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:39.642755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34399 (local address 127.0.0.1:45858)
0928 08:49:39.642912 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:39.643028 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 08:49:39.643539 22854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:34399: Network error: Client connection negotiation failed: client connection to 127.0.97.61:34399: connect: Connection refused (error 111)
W20260928 08:49:39.650982 22861 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 08:49:39.651355 22861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:49:39.651484 22861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:49:39.651629 22861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:49:39.659153 22861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:49:39.659276 22861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:49:39.659380 22861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:49:39.659477 22861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:49:39.659569 22861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:49:39.669865 22861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40259,127.0.97.61:34399,127.0.97.60:46481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:34399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:49:39.671291 22861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:49:39.672657 22861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:49:39.685042 22866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:39.684963 22867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:39.685074 22869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:49:39.685802 22861 server_base.cc:1034] running on GCE node
I20260928 08:49:39.686506 22861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:49:39.687903 22861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:49:39.689163 22861 hybrid_clock.cc:648] HybridClock initialized: now 1790585379689099 us; error 32 us; skew 500 ppm
I20260928 08:49:39.689551 22861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:49:39 dist-test-slave-n6xs krb5kdc[22774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:49:39.693020 22861 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260928 08:49:39.694573 22861 webserver.cc:460] Webserver started at http://127.0.97.61:33713/ using document root <none> and password file <none>
I20260928 08:49:39.695272 22861 fs_manager.cc:359] Metadata directory not provided
I20260928 08:49:39.695372 22861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:49:39.695672 22861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:49:39.699160 22861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-1/data/instance:
uuid: "3666d2df013c44bca63e7b7b1f142730"
format_stamp: "Formatted at 2026-09-28 08:49:39 on dist-test-slave-n6xs"
server_key: "748fab4d55e2ea27e486ce0909c57b58"
server_key_iv: "57b92e7dae9dcc7936c32577660462a4"
server_key_version: "encryptionkey@0"
I20260928 08:49:39.700023 22861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-1/wal/instance:
uuid: "3666d2df013c44bca63e7b7b1f142730"
format_stamp: "Formatted at 2026-09-28 08:49:39 on dist-test-slave-n6xs"
server_key: "748fab4d55e2ea27e486ce0909c57b58"
server_key_iv: "57b92e7dae9dcc7936c32577660462a4"
server_key_version: "encryptionkey@0"
I20260928 08:49:39.705389 22861 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 08:49:39.709087 22876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:49:39.710529 22861 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:49:39.710752 22861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-1/wal
uuid: "3666d2df013c44bca63e7b7b1f142730"
format_stamp: "Formatted at 2026-09-28 08:49:39 on dist-test-slave-n6xs"
server_key: "748fab4d55e2ea27e486ce0909c57b58"
server_key_iv: "57b92e7dae9dcc7936c32577660462a4"
server_key_version: "encryptionkey@0"
I20260928 08:49:39.711019 22861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:49:39.740629 22861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:49:39.745311 22861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:49:39.745608 22861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:49:39.764874 22861 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:34399
I20260928 08:49:39.764909 22927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:34399 every 8 connection(s)
I20260928 08:49:39.766320 22861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-1/data/info.pb
I20260928 08:49:39.766817 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 22861
I20260928 08:49:39.767037 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-1/wal/instance
I20260928 08:49:39.767659 388 external_mini_cluster.cc:1428] Setting key 5ea581677fc8c00dceace42323ef5172
I20260928 08:49:39.770500 22928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:49:39.777974 22928 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40259 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
Sep 28 08:49:39 dist-test-slave-n6xs krb5kdc[22774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:49:39.803573 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:39.781407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50838 (local address 127.0.97.62:40259)
0928 08:49:39.781970 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:39.781978 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:39.782944 (+ 966us) server_negotiation.cc:407] Connection header received
0928 08:49:39.784927 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:39.784936 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:39.785027 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:39.785168 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:39.786711 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:39.788050 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:39.789574 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:39.790282 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:39.790431 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:39.796175 (+ 5744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:39.796185 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:39.796193 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:39.796249 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:39.800058 (+ 3809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:39.800894 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:39.800903 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:39.800909 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:39.801063 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:39.801467 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:39.801475 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:39.801481 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:39.801903 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:39.802058 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:39.802804 (+ 746us) server_negotiation.cc:299] Negotiation successful
0928 08:49:39.803064 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":280,"threads_started":1}
I20260928 08:49:39.804313 22932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:39.781914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40259 (local address 127.0.0.1:50838)
0928 08:49:39.782754 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:39.782798 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:49:39.784631 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:49:39.785390 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:49:39.785398 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:49:39.785921 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:49:39.786479 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:39.786495 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:39.788214 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:39.788218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:39.789420 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:39.789431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:39.790491 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:39.790495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:39.790692 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:39.791393 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:49:39.791415 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:49:39.795888 (+ 4473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:49:39.800221 (+ 4333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:39.800228 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:39.800243 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:39.800758 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:39.801173 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:39.801176 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:39.801178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:39.801351 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:39.802037 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:49:39.802043 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:49:39.802426 (+ 383us) client_negotiation.cc:769] Sending connection context
0928 08:49:39.802676 (+ 250us) client_negotiation.cc:240] Negotiation successful
0928 08:49:39.803082 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":393,"threads_started":1}
I20260928 08:49:39.806445 22928 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34399 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:49:39 dist-test-slave-n6xs krb5kdc[22774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 08:49:39.822960 22932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:39.807269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34399 (local address 127.0.0.1:45872)
0928 08:49:39.807560 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:39.807582 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 08:49:39.807712 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:49:39.808508 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:49:39.808515 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:49:39.809058 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:49:39.809312 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:39.809328 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:39.810703 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:39.810712 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:39.812353 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:39.812372 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:39.813237 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:39.813250 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:39.813498 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:39.814189 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:49:39.814224 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:49:39.816977 (+ 2753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:49:39.820461 (+ 3484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:39.820469 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:39.820474 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:39.820834 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:39.821217 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:39.821222 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:39.821225 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:39.821346 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:39.821944 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:49:39.821949 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:49:39.822089 (+ 140us) client_negotiation.cc:769] Sending connection context
0928 08:49:39.822268 (+ 179us) client_negotiation.cc:240] Negotiation successful
0928 08:49:39.822567 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 08:49:39.823067 22934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:39.807365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45872 (local address 127.0.97.61:34399)
0928 08:49:39.808055 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:39.808061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:39.808087 (+ 26us) server_negotiation.cc:407] Connection header received
0928 08:49:39.808130 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:39.808136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:39.808324 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:39.808443 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:39.809498 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:39.810527 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:39.812496 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:39.813063 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:39.813161 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:39.817202 (+ 4041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:39.817214 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:39.817227 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:39.817256 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:39.820297 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:39.820965 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:39.820974 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:39.820984 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:39.821089 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:39.821455 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:39.821462 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:39.821467 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:39.821821 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:39.822043 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:39.822416 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 08:49:39.822691 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":545,"thread_start_us":377,"threads_started":1}
I20260928 08:49:39.825610 22928 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46481 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:49:39.826786 22932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:39.826290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46481 (local address 127.0.0.1:36962)
0928 08:49:39.826445 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:39.826629 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 08:49:39.827103 22878 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46481: connect: Connection refused (error 111)
I20260928 08:49:39.828790 22932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:39.828486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46481 (local address 127.0.0.1:36964)
0928 08:49:39.828609 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:39.828687 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 08:49:39.829059 22928 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46481: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46481: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260928 08:49:39.831681 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:46481
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:45619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:40259,127.0.97.61:34399,127.0.97.60:46481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:49:39.878425 22928 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46481 } attempt: 1
I20260928 08:49:39.879724 22932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:39.879152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46481 (local address 127.0.0.1:36978)
0928 08:49:39.879448 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:39.879564 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260928 08:49:39.880761 22932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:39.880444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46481 (local address 127.0.0.1:36990)
0928 08:49:39.880590 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:39.880660 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 08:49:39.881047 22928 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46481: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46481: connect: Connection refused (error 111)
I20260928 08:49:39.942878 22854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34399 } attempt: 5
I20260928 08:49:39.949357 22928 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46481 } attempt: 2
I20260928 08:49:39.950281 22932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:39.949869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46481 (local address 127.0.0.1:37006)
0928 08:49:39.950044 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:39.950137 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 08:49:39.951339 22932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:39.950937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46481 (local address 127.0.0.1:37012)
0928 08:49:39.951121 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:39.951206 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
Sep 28 08:49:39 dist-test-slave-n6xs krb5kdc[22774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
W20260928 08:49:39.952083 22928 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46481: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46481: connect: Connection refused (error 111)
I20260928 08:49:39.958005 22934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:39.943994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45886 (local address 127.0.97.61:34399)
0928 08:49:39.944179 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:39.944183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:39.944203 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:49:39.944279 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:39.944283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:39.944343 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:39.944429 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:39.945517 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:39.946244 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:39.948150 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:39.948628 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:39.948732 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:39.952713 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:39.952718 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:39.952722 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:39.952752 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:39.955378 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:39.956089 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:39.956097 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:39.956105 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:39.956219 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:39.956612 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:39.956623 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:39.956625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:39.956894 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:39.957051 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:39.957475 (+ 424us) server_negotiation.cc:299] Negotiation successful
0928 08:49:39.957695 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 08:49:39.958022 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:39.943708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34399 (local address 127.0.0.1:45886)
0928 08:49:39.943975 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:39.944000 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 08:49:39.944160 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:49:39.944509 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:49:39.944514 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:49:39.945077 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:49:39.945335 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:39.945350 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:39.946420 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:39.946426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:39.947927 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:39.947947 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:39.948796 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:39.948804 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:39.949010 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:39.949798 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:49:39.949826 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:49:39.952521 (+ 2695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:49:39.955514 (+ 2993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:39.955522 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:39.955526 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:39.955968 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:39.956337 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:39.956342 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:39.956346 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:39.956491 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:39.957010 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:49:39.957014 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:49:39.957195 (+ 181us) client_negotiation.cc:769] Sending connection context
0928 08:49:39.957357 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 08:49:39.957668 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260928 08:49:39.959132 22854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46481 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:49:39.960296 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:39.959927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46481 (local address 127.0.0.1:37026)
0928 08:49:39.960095 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:39.960165 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 08:49:39.961185 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:39.960861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46481 (local address 127.0.0.1:37040)
0928 08:49:39.961001 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:39.961083 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 08:49:39.961474 22854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46481: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46481: connect: Connection refused (error 111)
I20260928 08:49:40.012795 22854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46481 } attempt: 1
I20260928 08:49:40.014151 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:40.013608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46481 (local address 127.0.0.1:37056)
0928 08:49:40.013809 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:40.013918 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 08:49:40.015341 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:40.014859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46481 (local address 127.0.0.1:37070)
0928 08:49:40.015066 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:40.015163 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 08:49:40.015686 22854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46481: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46481: connect: Connection refused (error 111)
I20260928 08:49:40.043349 22928 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46481 } attempt: 3
I20260928 08:49:40.044623 22932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:40.044155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46481 (local address 127.0.0.1:37082)
0928 08:49:40.044382 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:40.044467 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 08:49:40.045681 22932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:40.045277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46481 (local address 127.0.0.1:37098)
0928 08:49:40.045496 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:40.045569 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260928 08:49:40.045962 22928 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46481: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46481: connect: Connection refused (error 111)
I20260928 08:49:40.083964 22854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46481 } attempt: 2
I20260928 08:49:40.085254 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:40.084609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46481 (local address 127.0.0.1:37102)
0928 08:49:40.084885 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:40.085042 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 08:49:40.086472 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:40.086021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46481 (local address 127.0.0.1:37114)
0928 08:49:40.086221 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:40.086316 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 08:49:40.086779 22854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46481: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46481: connect: Connection refused (error 111)
W20260928 08:49:40.180053 22936 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 08:49:40.180401 22936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:49:40.180500 22936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:49:40.180630 22936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:49:40.188033 22936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:49:40.188130 22936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:49:40.188211 22936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:49:40.188287 22936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:49:40.188382 22936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:49:40.189178 22928 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46481 } attempt: 4
I20260928 08:49:40.190371 22932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:40.189841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46481 (local address 127.0.0.1:37124)
0928 08:49:40.190079 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:40.190185 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 08:49:40.191532 22932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:40.191155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46481 (local address 127.0.0.1:37130)
0928 08:49:40.191321 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:40.191401 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 08:49:40.191776 22928 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46481: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46481: connect: Connection refused (error 111)
I20260928 08:49:40.193009 22854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46481 } attempt: 3
I20260928 08:49:40.193993 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:40.193547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46481 (local address 127.0.0.1:37136)
0928 08:49:40.193732 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:40.193824 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 08:49:40.195055 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:40.194664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46481 (local address 127.0.0.1:37140)
0928 08:49:40.194854 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:40.194931 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 08:49:40.195302 22854 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:46481: Network error: Client connection negotiation failed: client connection to 127.0.97.60:46481: connect: Connection refused (error 111)
I20260928 08:49:40.199914 22936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:45619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:40259,127.0.97.61:34399,127.0.97.60:46481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:46481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:49:40.201206 22936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:49:40.202462 22936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:49:40.213578 22941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:40.214171 22944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:40.214274 22942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:49:40.214890 22936 server_base.cc:1034] running on GCE node
I20260928 08:49:40.215919 22936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:49:40.217752 22936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:49:40.218986 22936 hybrid_clock.cc:648] HybridClock initialized: now 1790585380218919 us; error 42 us; skew 500 ppm
I20260928 08:49:40.219254 22936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:49:40 dist-test-slave-n6xs krb5kdc[22774](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:49:40.222324 22936 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260928 08:49:40.223730 22936 webserver.cc:460] Webserver started at http://127.0.97.60:40177/ using document root <none> and password file <none>
I20260928 08:49:40.224407 22936 fs_manager.cc:359] Metadata directory not provided
I20260928 08:49:40.224512 22936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:49:40.224822 22936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:49:40.227715 22936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-2/data/instance:
uuid: "1e806cc37e1a4ca38c37aa570874bb9d"
format_stamp: "Formatted at 2026-09-28 08:49:40 on dist-test-slave-n6xs"
server_key: "0da23af17d2945739aa552dffaaefa27"
server_key_iv: "f4f5de490c32c622cb43fb7e0abacc16"
server_key_version: "encryptionkey@0"
I20260928 08:49:40.228524 22936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-2/wal/instance:
uuid: "1e806cc37e1a4ca38c37aa570874bb9d"
format_stamp: "Formatted at 2026-09-28 08:49:40 on dist-test-slave-n6xs"
server_key: "0da23af17d2945739aa552dffaaefa27"
server_key_iv: "f4f5de490c32c622cb43fb7e0abacc16"
server_key_version: "encryptionkey@0"
I20260928 08:49:40.233700 22936 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:49:40.238667 22951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:49:40.240429 22936 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260928 08:49:40.240631 22936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-2/wal
uuid: "1e806cc37e1a4ca38c37aa570874bb9d"
format_stamp: "Formatted at 2026-09-28 08:49:40 on dist-test-slave-n6xs"
server_key: "0da23af17d2945739aa552dffaaefa27"
server_key_iv: "f4f5de490c32c622cb43fb7e0abacc16"
server_key_version: "encryptionkey@0"
I20260928 08:49:40.240914 22936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:49:40.258749 22936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:49:40.263319 22936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:49:40.263628 22936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:49:40.283557 22936 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:46481
I20260928 08:49:40.283602 23002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:46481 every 8 connection(s)
I20260928 08:49:40.285050 22936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-2/data/info.pb
I20260928 08:49:40.288309 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 22936
I20260928 08:49:40.288509 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790585022442112-388-0/minicluster-data/master-2/wal/instance
I20260928 08:49:40.288986 23003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:49:40.289142 388 external_mini_cluster.cc:1428] Setting key 278810db57036f59b08f78f5d084d00d
I20260928 08:49:40.296554 23003 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40259 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:49:40 dist-test-slave-n6xs krb5kdc[22774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 28 08:49:40 dist-test-slave-n6xs krb5kdc[22774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260928 08:49:40.317974 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:40.300832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50850 (local address 127.0.97.62:40259)
0928 08:49:40.301065 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:40.301080 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:40.302069 (+ 989us) server_negotiation.cc:407] Connection header received
0928 08:49:40.302332 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:40.302341 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:40.302432 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:40.302502 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:40.304054 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:40.304693 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:40.306220 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:40.307070 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:40.307228 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:40.310513 (+ 3285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:40.310524 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:40.310541 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:40.310599 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:40.313844 (+ 3245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:40.315012 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:40.315020 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:40.315023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:40.315170 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:40.315658 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:40.315665 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:40.315668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:40.316028 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:40.316182 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:40.317143 (+ 961us) server_negotiation.cc:299] Negotiation successful
0928 08:49:40.317452 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 08:49:40.319172 23007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:40.301199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:40259 (local address 127.0.0.1:50850)
0928 08:49:40.301950 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:40.301991 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 08:49:40.302186 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:49:40.302745 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:49:40.302751 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:49:40.303547 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:49:40.303918 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:40.303927 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:40.304856 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:40.304861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:40.306055 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:40.306066 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:40.307332 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:40.307341 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:40.307554 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:40.308157 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:49:40.308177 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:49:40.310210 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:49:40.314196 (+ 3986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:40.314205 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:40.314218 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:40.314900 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:40.315283 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:40.315288 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:40.315292 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:40.315550 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:40.316154 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:49:40.316162 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:49:40.316622 (+ 460us) client_negotiation.cc:769] Sending connection context
0928 08:49:40.316969 (+ 347us) client_negotiation.cc:240] Negotiation successful
0928 08:49:40.317498 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":324,"threads_started":1}
I20260928 08:49:40.320677 23006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:40.291879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37156 (local address 127.0.97.60:46481)
0928 08:49:40.296349 (+ 4470us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:40.296380 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:40.296432 (+ 52us) server_negotiation.cc:407] Connection header received
0928 08:49:40.298625 (+ 2193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:40.298664 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:40.299132 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:40.300305 (+ 1173us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:40.301416 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:40.303469 (+ 2053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:40.305891 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:40.306508 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:40.306708 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:40.311903 (+ 5195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:40.311919 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:40.311942 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:40.311996 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:40.316613 (+ 4617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:40.317482 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:40.317490 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:40.317493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:40.317650 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:40.318079 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:40.318085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:40.318087 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:40.318454 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:40.318776 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:40.320115 (+ 1339us) server_negotiation.cc:299] Negotiation successful
0928 08:49:40.320285 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4224,"thread_start_us":3973,"threads_started":1}
I20260928 08:49:40.321162 23003 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34399 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:49:40 dist-test-slave-n6xs krb5kdc[22774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
Sep 28 08:49:40 dist-test-slave-n6xs krb5kdc[22774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:49:40.337186 22934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:40.322661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45900 (local address 127.0.97.61:34399)
0928 08:49:40.322892 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:40.322900 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:40.322920 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:49:40.322988 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:40.322996 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:40.323090 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:40.323218 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:40.323853 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:40.324820 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:40.326033 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:40.326476 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:40.326610 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:40.329557 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:40.329561 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:40.329565 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:40.329602 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:40.333745 (+ 4143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:40.334559 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:40.334569 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:40.334575 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:40.334806 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:40.335422 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:40.335428 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:40.335431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:40.335781 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:40.335993 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:40.336506 (+ 513us) server_negotiation.cc:299] Negotiation successful
0928 08:49:40.336724 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 08:49:40.337802 23007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:40.322112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:34399 (local address 127.0.0.1:45900)
0928 08:49:40.322315 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:40.322333 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 08:49:40.322423 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:49:40.323233 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:49:40.323236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:49:40.323529 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:49:40.323657 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:40.323664 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:40.325008 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:40.325013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:40.325887 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:40.325897 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:40.326623 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:40.326628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:40.326801 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:40.327385 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:49:40.327400 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:49:40.329370 (+ 1970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:49:40.333910 (+ 4540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:40.333916 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:40.333920 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:40.334415 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:40.335023 (+ 608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:40.335031 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:40.335035 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:40.335277 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:40.335967 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:49:40.335975 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:49:40.336179 (+ 204us) client_negotiation.cc:769] Sending connection context
0928 08:49:40.336374 (+ 195us) client_negotiation.cc:240] Negotiation successful
0928 08:49:40.337020 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 08:49:40.339028 23003 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46481 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:49:40.341594 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:40.324476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50854 (local address 127.0.97.62:40259)
0928 08:49:40.324636 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:40.324641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:40.324659 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:49:40.324705 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:40.324709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:40.324779 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:40.324879 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:40.326826 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:40.328159 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:40.330374 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:40.330808 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:40.330951 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:40.335023 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:40.335028 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:40.335032 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:40.335062 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:40.338268 (+ 3206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:40.339064 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:40.339070 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:40.339074 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:40.339213 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:40.339585 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:40.339591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:40.339594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:40.339956 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:40.340408 (+ 452us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:40.341204 (+ 796us) server_negotiation.cc:299] Negotiation successful
0928 08:49:40.341359 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 08:49:40 dist-test-slave-n6xs krb5kdc[22774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585380, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 28 08:49:40 dist-test-slave-n6xs krb5kdc[22774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585378, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 08:49:40.356534 23007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:40.339869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46481 (local address 127.0.0.1:37158)
0928 08:49:40.340229 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:40.340253 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 08:49:40.340413 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:49:40.340757 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:49:40.340769 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:49:40.341334 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:49:40.341588 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:40.341604 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:40.343213 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:40.343222 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:40.344973 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:40.344992 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:40.346481 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:40.346490 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:40.346756 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:40.347435 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:49:40.347468 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:49:40.350645 (+ 3177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:49:40.353964 (+ 3319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:40.353971 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:40.353975 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:40.354326 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:40.354678 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:40.354685 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:40.354690 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:40.354816 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:40.355500 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:49:40.355504 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:49:40.355645 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 08:49:40.355794 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 08:49:40.356129 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221}
I20260928 08:49:40.356563 23006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:40.339938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37158 (local address 127.0.97.60:46481)
0928 08:49:40.340077 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:40.340084 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:40.340319 (+ 235us) server_negotiation.cc:407] Connection header received
0928 08:49:40.340530 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:40.340535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:40.340618 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:40.340707 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:40.341765 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:40.343020 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:40.345191 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:40.346017 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:40.346148 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:40.350858 (+ 4710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:40.350866 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:40.350871 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:40.350907 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:40.353836 (+ 2929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:40.354441 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:40.354446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:40.354456 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:40.354575 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:40.355128 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:40.355132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:40.355134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:40.355388 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:40.355536 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:40.355912 (+ 376us) server_negotiation.cc:299] Negotiation successful
0928 08:49:40.356275 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":150,"server-negotiator.queue_time_us":34}
I20260928 08:49:40.360579 22934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:40.343409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45916 (local address 127.0.97.61:34399)
0928 08:49:40.343579 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:40.343587 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:40.343605 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:49:40.343643 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:40.343646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:40.343695 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:40.343796 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:40.345028 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:40.345957 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:40.347416 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:40.347913 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:40.348054 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:40.352348 (+ 4294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:40.352358 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:40.352368 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:40.352422 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:40.357278 (+ 4856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:40.358144 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:40.358150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:40.358152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:40.358313 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:40.358914 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:40.358919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:40.358922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:40.359300 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:40.359587 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:40.360009 (+ 422us) server_negotiation.cc:299] Negotiation successful
0928 08:49:40.360238 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 08:49:40.362792 23003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e806cc37e1a4ca38c37aa570874bb9d: Bootstrap starting.
I20260928 08:49:40.366379 23003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e806cc37e1a4ca38c37aa570874bb9d: Neither blocks nor log segments found. Creating new log.
I20260928 08:49:40.366837 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:49:40.367519 23003 log.cc:824] T 00000000000000000000000000000000 P 1e806cc37e1a4ca38c37aa570874bb9d: Log is configured to *not* fsync() on all Append() calls
I20260928 08:49:40.370169 23003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e806cc37e1a4ca38c37aa570874bb9d: No bootstrap required, opened a new log
I20260928 08:49:40.375908 22854 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46481 } attempt: 4
I20260928 08:49:40.371122 22829 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "8366fd63aacf45bdb7bcb6f9aa55c00f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50854
I20260928 08:49:40.377939 23003 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e806cc37e1a4ca38c37aa570874bb9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8366fd63aacf45bdb7bcb6f9aa55c00f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40259 } } peers { permanent_uuid: "3666d2df013c44bca63e7b7b1f142730" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34399 } } peers { permanent_uuid: "1e806cc37e1a4ca38c37aa570874bb9d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46481 } }
I20260928 08:49:40.378252 23003 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e806cc37e1a4ca38c37aa570874bb9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e806cc37e1a4ca38c37aa570874bb9d, State: Initialized, Role: FOLLOWER
I20260928 08:49:40.378818 23003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e806cc37e1a4ca38c37aa570874bb9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8366fd63aacf45bdb7bcb6f9aa55c00f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40259 } } peers { permanent_uuid: "3666d2df013c44bca63e7b7b1f142730" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34399 } } peers { permanent_uuid: "1e806cc37e1a4ca38c37aa570874bb9d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46481 } }
I20260928 08:49:40.380033 23009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e806cc37e1a4ca38c37aa570874bb9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8366fd63aacf45bdb7bcb6f9aa55c00f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40259 } } peers { permanent_uuid: "3666d2df013c44bca63e7b7b1f142730" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34399 } } peers { permanent_uuid: "1e806cc37e1a4ca38c37aa570874bb9d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46481 } } }
I20260928 08:49:40.380558 23009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e806cc37e1a4ca38c37aa570874bb9d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:49:40.381237 23003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e806cc37e1a4ca38c37aa570874bb9d [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 08:49:40 dist-test-slave-n6xs krb5kdc[22774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260928 08:49:40.396870 22857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:40.376621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46481 (local address 127.0.0.1:37172)
0928 08:49:40.376918 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:40.376942 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 08:49:40.377073 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:49:40.377516 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:49:40.377523 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:49:40.378153 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:49:40.378433 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:40.378449 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:40.380023 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:40.380028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:40.381404 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:40.381419 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:40.382842 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:40.382852 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:40.383141 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:40.384067 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:49:40.384102 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:49:40.387903 (+ 3801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:49:40.393383 (+ 5480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:40.393391 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:40.393398 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:40.393980 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:40.394464 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:40.394489 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:40.394495 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:40.394704 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:40.395452 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:49:40.395459 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:49:40.395677 (+ 218us) client_negotiation.cc:769] Sending connection context
0928 08:49:40.395893 (+ 216us) client_negotiation.cc:240] Negotiation successful
0928 08:49:40.396309 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260928 08:49:40.398085 23006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:40.376723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37172 (local address 127.0.97.60:46481)
0928 08:49:40.376886 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:40.376893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:40.377194 (+ 301us) server_negotiation.cc:407] Connection header received
0928 08:49:40.377270 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:40.377275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:40.377344 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:40.377465 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:40.378609 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:40.379803 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:40.381565 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:40.382604 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:40.383626 (+ 1022us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:40.388152 (+ 4526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:40.388164 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:40.388172 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:40.388233 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:40.393243 (+ 5010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:40.394129 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:40.394136 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:40.394142 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:40.394336 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:40.394828 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:40.394835 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:40.394841 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:40.395317 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:40.395516 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:40.397239 (+ 1723us) server_negotiation.cc:299] Negotiation successful
0928 08:49:40.397570 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 08:49:40.405030 22854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8366fd63aacf45bdb7bcb6f9aa55c00f: Bootstrap starting.
I20260928 08:49:40.409899 22854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8366fd63aacf45bdb7bcb6f9aa55c00f: Neither blocks nor log segments found. Creating new log.
I20260928 08:49:40.411707 22854 log.cc:824] T 00000000000000000000000000000000 P 8366fd63aacf45bdb7bcb6f9aa55c00f: Log is configured to *not* fsync() on all Append() calls
I20260928 08:49:40.415776 22854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8366fd63aacf45bdb7bcb6f9aa55c00f: No bootstrap required, opened a new log
I20260928 08:49:40.423548 22854 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8366fd63aacf45bdb7bcb6f9aa55c00f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8366fd63aacf45bdb7bcb6f9aa55c00f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40259 } } peers { permanent_uuid: "3666d2df013c44bca63e7b7b1f142730" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34399 } } peers { permanent_uuid: "1e806cc37e1a4ca38c37aa570874bb9d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46481 } }
I20260928 08:49:40.424091 22854 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8366fd63aacf45bdb7bcb6f9aa55c00f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8366fd63aacf45bdb7bcb6f9aa55c00f, State: Initialized, Role: FOLLOWER
I20260928 08:49:40.424665 22854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8366fd63aacf45bdb7bcb6f9aa55c00f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8366fd63aacf45bdb7bcb6f9aa55c00f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40259 } } peers { permanent_uuid: "3666d2df013c44bca63e7b7b1f142730" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34399 } } peers { permanent_uuid: "1e806cc37e1a4ca38c37aa570874bb9d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46481 } }
I20260928 08:49:40.416234 22936 master_runner.cc:186] Error getting master registration for 127.0.97.62:40259: OK, instance_id { permanent_uuid: "8366fd63aacf45bdb7bcb6f9aa55c00f" instance_seqno: 1790585379227764 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 08:49:40.428193 22936 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 08:49:40.434643 23021 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1e806cc37e1a4ca38c37aa570874bb9d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:49:40.434777 23021 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:49:40.440836 23023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8366fd63aacf45bdb7bcb6f9aa55c00f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8366fd63aacf45bdb7bcb6f9aa55c00f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40259 } } peers { permanent_uuid: "3666d2df013c44bca63e7b7b1f142730" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34399 } } peers { permanent_uuid: "1e806cc37e1a4ca38c37aa570874bb9d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46481 } } }
I20260928 08:49:40.441633 23023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8366fd63aacf45bdb7bcb6f9aa55c00f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:49:40.441959 22854 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8366fd63aacf45bdb7bcb6f9aa55c00f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:49:40.442065 22829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8366fd63aacf45bdb7bcb6f9aa55c00f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 08:49:40.442255 22829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8366fd63aacf45bdb7bcb6f9aa55c00f [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:49:40.446672 22829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8366fd63aacf45bdb7bcb6f9aa55c00f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8366fd63aacf45bdb7bcb6f9aa55c00f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40259 } } peers { permanent_uuid: "3666d2df013c44bca63e7b7b1f142730" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34399 } } peers { permanent_uuid: "1e806cc37e1a4ca38c37aa570874bb9d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46481 } }
I20260928 08:49:40.450872 22978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8366fd63aacf45bdb7bcb6f9aa55c00f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1e806cc37e1a4ca38c37aa570874bb9d"
I20260928 08:49:40.451478 22978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e806cc37e1a4ca38c37aa570874bb9d [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:49:40.456581 22978 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1e806cc37e1a4ca38c37aa570874bb9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8366fd63aacf45bdb7bcb6f9aa55c00f in term 1.
I20260928 08:49:40.457743 22804 leader_election.cc:302] T 00000000000000000000000000000000 P 8366fd63aacf45bdb7bcb6f9aa55c00f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e806cc37e1a4ca38c37aa570874bb9d, 8366fd63aacf45bdb7bcb6f9aa55c00f; no voters:
I20260928 08:49:40.458539 22829 leader_election.cc:288] T 00000000000000000000000000000000 P 8366fd63aacf45bdb7bcb6f9aa55c00f [CANDIDATE]: Term 1 election: Requested vote from peers 3666d2df013c44bca63e7b7b1f142730 (127.0.97.61:34399), 1e806cc37e1a4ca38c37aa570874bb9d (127.0.97.60:46481)
W20260928 08:49:40.467198 22808 tablet.cc:2369] T 00000000000000000000000000000000 P 8366fd63aacf45bdb7bcb6f9aa55c00f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:49:40.468402 23023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8366fd63aacf45bdb7bcb6f9aa55c00f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:49:40.478016 23023 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8366fd63aacf45bdb7bcb6f9aa55c00f [term 1 LEADER]: Becoming Leader. State: Replica: 8366fd63aacf45bdb7bcb6f9aa55c00f, State: Running, Role: LEADER
I20260928 08:49:40.482712 23023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8366fd63aacf45bdb7bcb6f9aa55c00f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8366fd63aacf45bdb7bcb6f9aa55c00f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40259 } } peers { permanent_uuid: "3666d2df013c44bca63e7b7b1f142730" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34399 } } peers { permanent_uuid: "1e806cc37e1a4ca38c37aa570874bb9d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46481 } }
I20260928 08:49:40.492117 22928 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46481 } attempt: 5
Sep 28 08:49:40 dist-test-slave-n6xs krb5kdc[22774](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585379, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260928 08:49:40.512506 22932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:40.492642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:46481 (local address 127.0.0.1:37180)
0928 08:49:40.492987 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:40.493008 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 08:49:40.493135 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:49:40.494151 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:49:40.494156 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:49:40.494697 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:49:40.494905 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:40.494916 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:40.496436 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:40.496442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:40.498204 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:40.498222 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:40.499736 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:40.499743 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:40.500037 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:40.500844 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:49:40.500876 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:49:40.504002 (+ 3126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:49:40.508941 (+ 4939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:40.508949 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:40.508956 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:40.509625 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:40.510164 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:40.510172 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:40.510178 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:40.510376 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:40.511177 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:49:40.511183 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:49:40.511380 (+ 197us) client_negotiation.cc:769] Sending connection context
0928 08:49:40.511558 (+ 178us) client_negotiation.cc:240] Negotiation successful
0928 08:49:40.511943 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246}
I20260928 08:49:40.513671 23006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:40.493469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37180 (local address 127.0.97.60:46481)
0928 08:49:40.493642 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:40.493648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:40.493668 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:49:40.493735 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:40.493741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:40.493825 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:40.493917 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:40.495080 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:40.496258 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:40.498458 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:40.499350 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:40.499489 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:40.504224 (+ 4735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:40.504233 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:40.504239 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:40.504278 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:40.508772 (+ 4494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:40.509788 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:40.509794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:40.509799 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:40.509993 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:40.510519 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:40.510525 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:40.510530 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:40.510931 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:40.511107 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:40.512967 (+ 1860us) server_negotiation.cc:299] Negotiation successful
0928 08:49:40.513204 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260928 08:49:40.515964 22957 tablet.cc:2369] T 00000000000000000000000000000000 P 1e806cc37e1a4ca38c37aa570874bb9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 08:49:40.520877 22928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3666d2df013c44bca63e7b7b1f142730: Bootstrap starting.
I20260928 08:49:40.525629 22928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3666d2df013c44bca63e7b7b1f142730: Neither blocks nor log segments found. Creating new log.
I20260928 08:49:40.527487 22928 log.cc:824] T 00000000000000000000000000000000 P 3666d2df013c44bca63e7b7b1f142730: Log is configured to *not* fsync() on all Append() calls
I20260928 08:49:40.534533 22928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3666d2df013c44bca63e7b7b1f142730: No bootstrap required, opened a new log
I20260928 08:49:40.534439 23026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8366fd63aacf45bdb7bcb6f9aa55c00f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8366fd63aacf45bdb7bcb6f9aa55c00f. Latest consensus state: current_term: 1 leader_uuid: "8366fd63aacf45bdb7bcb6f9aa55c00f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8366fd63aacf45bdb7bcb6f9aa55c00f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40259 } } peers { permanent_uuid: "3666d2df013c44bca63e7b7b1f142730" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34399 } } peers { permanent_uuid: "1e806cc37e1a4ca38c37aa570874bb9d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46481 } } }
I20260928 08:49:40.534929 23026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8366fd63aacf45bdb7bcb6f9aa55c00f [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:40.551010 23037 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 08:49:40.568118 23037 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:49:40.583308 22934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:40.488634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45926 (local address 127.0.97.61:34399)
0928 08:49:40.488910 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:40.488919 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:40.504835 (+ 15916us) server_negotiation.cc:407] Connection header received
0928 08:49:40.515950 (+ 11115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:40.515956 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:40.516037 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:40.516108 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:40.543467 (+ 27359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:40.544559 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:40.553580 (+ 9021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:40.555216 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:40.555339 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:40.573719 (+ 18380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:40.573726 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:40.573731 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:40.573793 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:40.578961 (+ 5168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:40.580068 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:40.580075 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:40.580080 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:40.580299 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:40.580843 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:40.580849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:40.580851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:40.581262 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:40.581670 (+ 408us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:40.582404 (+ 734us) server_negotiation.cc:299] Negotiation successful
0928 08:49:40.582766 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 08:49:40.592850 22928 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3666d2df013c44bca63e7b7b1f142730 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8366fd63aacf45bdb7bcb6f9aa55c00f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40259 } } peers { permanent_uuid: "3666d2df013c44bca63e7b7b1f142730" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34399 } } peers { permanent_uuid: "1e806cc37e1a4ca38c37aa570874bb9d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46481 } }
I20260928 08:49:40.593269 22928 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3666d2df013c44bca63e7b7b1f142730 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3666d2df013c44bca63e7b7b1f142730, State: Initialized, Role: FOLLOWER
I20260928 08:49:40.593812 22928 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3666d2df013c44bca63e7b7b1f142730 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8366fd63aacf45bdb7bcb6f9aa55c00f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40259 } } peers { permanent_uuid: "3666d2df013c44bca63e7b7b1f142730" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34399 } } peers { permanent_uuid: "1e806cc37e1a4ca38c37aa570874bb9d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46481 } }
I20260928 08:49:40.597808 22978 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1e806cc37e1a4ca38c37aa570874bb9d [term 1 FOLLOWER]: Refusing update from remote peer 8366fd63aacf45bdb7bcb6f9aa55c00f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:49:40.599226 23026 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8366fd63aacf45bdb7bcb6f9aa55c00f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e806cc37e1a4ca38c37aa570874bb9d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:49:40.608385 23043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3666d2df013c44bca63e7b7b1f142730 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8366fd63aacf45bdb7bcb6f9aa55c00f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40259 } } peers { permanent_uuid: "3666d2df013c44bca63e7b7b1f142730" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34399 } } peers { permanent_uuid: "1e806cc37e1a4ca38c37aa570874bb9d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46481 } } }
I20260928 08:49:40.608844 23043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3666d2df013c44bca63e7b7b1f142730 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:49:40.609604 23006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:40.505639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37196 (local address 127.0.97.60:46481)
0928 08:49:40.526012 (+ 20373us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:40.526021 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:40.539724 (+ 13703us) server_negotiation.cc:407] Connection header received
0928 08:49:40.541302 (+ 1578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:40.541308 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:40.541417 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:40.541499 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:40.550910 (+ 9411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:40.552003 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:40.562987 (+ 10984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:40.563718 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:40.563834 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:40.584679 (+ 20845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:40.584688 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:40.584697 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:40.584747 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:40.596081 (+ 11334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:40.605533 (+ 9452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:40.605541 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:40.605544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:40.605734 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:40.606294 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:40.606300 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:40.606303 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:40.606704 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:40.606984 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:40.608170 (+ 1186us) server_negotiation.cc:299] Negotiation successful
0928 08:49:40.608460 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20222,"thread_start_us":25693,"threads_started":1}
I20260928 08:49:40.611372 22928 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3666d2df013c44bca63e7b7b1f142730 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:49:40.611781 22902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3666d2df013c44bca63e7b7b1f142730 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:49:40.615034 22931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:40.490105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50870 (local address 127.0.97.62:40259)
0928 08:49:40.490244 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:40.490252 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:40.515238 (+ 24986us) server_negotiation.cc:407] Connection header received
0928 08:49:40.517558 (+ 2320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:40.517564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:40.517663 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:40.517730 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:40.543314 (+ 25584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:40.544524 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:40.559731 (+ 15207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:40.560639 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:40.560738 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:40.585676 (+ 24938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:40.585683 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:40.585689 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:40.585732 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:40.591428 (+ 5696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:40.612013 (+ 20585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:40.612022 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:40.612026 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:40.612794 (+ 768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:40.613243 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:40.613251 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:40.613256 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:40.613634 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:40.613886 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:40.614439 (+ 553us) server_negotiation.cc:299] Negotiation successful
0928 08:49:40.614680 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 08:49:40.616129 22903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8366fd63aacf45bdb7bcb6f9aa55c00f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3666d2df013c44bca63e7b7b1f142730"
I20260928 08:49:40.616607 22902 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3666d2df013c44bca63e7b7b1f142730 [term 1 FOLLOWER]: Refusing update from remote peer 8366fd63aacf45bdb7bcb6f9aa55c00f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:49:40.617847 23026 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8366fd63aacf45bdb7bcb6f9aa55c00f [LEADER]: Connected to new peer: Peer: permanent_uuid: "3666d2df013c44bca63e7b7b1f142730" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:49:40.621382 22903 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3666d2df013c44bca63e7b7b1f142730 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8366fd63aacf45bdb7bcb6f9aa55c00f in term 1.
I20260928 08:49:40.632031 23053 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 08:49:40.635389 23009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e806cc37e1a4ca38c37aa570874bb9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8366fd63aacf45bdb7bcb6f9aa55c00f. Latest consensus state: current_term: 1 leader_uuid: "8366fd63aacf45bdb7bcb6f9aa55c00f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8366fd63aacf45bdb7bcb6f9aa55c00f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40259 } } peers { permanent_uuid: "3666d2df013c44bca63e7b7b1f142730" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34399 } } peers { permanent_uuid: "1e806cc37e1a4ca38c37aa570874bb9d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46481 } } }
I20260928 08:49:40.635671 23009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e806cc37e1a4ca38c37aa570874bb9d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:49:40.648236 23023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8366fd63aacf45bdb7bcb6f9aa55c00f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8366fd63aacf45bdb7bcb6f9aa55c00f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8366fd63aacf45bdb7bcb6f9aa55c00f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40259 } } peers { permanent_uuid: "3666d2df013c44bca63e7b7b1f142730" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34399 } } peers { permanent_uuid: "1e806cc37e1a4ca38c37aa570874bb9d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46481 } } }
I20260928 08:49:40.648542 23023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8366fd63aacf45bdb7bcb6f9aa55c00f [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:40.635952 22787 master_runner.cc:186] Error getting master registration for 127.0.97.61:34399: OK, instance_id { permanent_uuid: "3666d2df013c44bca63e7b7b1f142730" instance_seqno: 1790585379759162 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 08:49:40.651223 23009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e806cc37e1a4ca38c37aa570874bb9d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8366fd63aacf45bdb7bcb6f9aa55c00f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8366fd63aacf45bdb7bcb6f9aa55c00f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40259 } } peers { permanent_uuid: "3666d2df013c44bca63e7b7b1f142730" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34399 } } peers { permanent_uuid: "1e806cc37e1a4ca38c37aa570874bb9d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46481 } } }
I20260928 08:49:40.651547 23009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e806cc37e1a4ca38c37aa570874bb9d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:49:40.652647 22787 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 08:49:40.653460 23037 catalog_manager.cc:1347] Generated new cluster ID: 560e479e0d914960887f98c95a4de2a2
I20260928 08:49:40.653537 23037 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:49:40.659891 23043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3666d2df013c44bca63e7b7b1f142730 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8366fd63aacf45bdb7bcb6f9aa55c00f. Latest consensus state: current_term: 1 leader_uuid: "8366fd63aacf45bdb7bcb6f9aa55c00f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8366fd63aacf45bdb7bcb6f9aa55c00f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40259 } } peers { permanent_uuid: "3666d2df013c44bca63e7b7b1f142730" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34399 } } peers { permanent_uuid: "1e806cc37e1a4ca38c37aa570874bb9d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46481 } } }
I20260928 08:49:40.660193 23043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3666d2df013c44bca63e7b7b1f142730 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:49:40.661412 23026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8366fd63aacf45bdb7bcb6f9aa55c00f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8366fd63aacf45bdb7bcb6f9aa55c00f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8366fd63aacf45bdb7bcb6f9aa55c00f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40259 } } peers { permanent_uuid: "3666d2df013c44bca63e7b7b1f142730" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34399 } } peers { permanent_uuid: "1e806cc37e1a4ca38c37aa570874bb9d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46481 } } }
I20260928 08:49:40.661671 23026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8366fd63aacf45bdb7bcb6f9aa55c00f [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:40.669277 23051 mvcc.cc:205] Tried to move back new op lower bound from 7334237718914613248 to 7334237718505295872. Current Snapshot: MvccSnapshot[applied={T|T < 7334237718914613248}]
I20260928 08:49:40.674394 23043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3666d2df013c44bca63e7b7b1f142730 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8366fd63aacf45bdb7bcb6f9aa55c00f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8366fd63aacf45bdb7bcb6f9aa55c00f" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 40259 } } peers { permanent_uuid: "3666d2df013c44bca63e7b7b1f142730" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 34399 } } peers { permanent_uuid: "1e806cc37e1a4ca38c37aa570874bb9d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 46481 } } }
I20260928 08:49:40.674659 23043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3666d2df013c44bca63e7b7b1f142730 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:49:40.685020 23037 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:49:40.686878 23037 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:49:40.712877 23037 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8366fd63aacf45bdb7bcb6f9aa55c00f: Generated new TSK 0
I20260928 08:49:40.714035 23037 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:49:40.734647 23068 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260928 08:49:40.806548 22931 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:49:40.796574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50872 (local address 127.0.97.62:40259)
0928 08:49:40.796967 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:40.796974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:40.804431 (+ 7457us) server_negotiation.cc:407] Connection header received
0928 08:49:40.804767 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:40.804775 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:40.804868 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:40.804953 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:40.806270 (+ 1317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50872: BlockingRecv error: recv got EOF from 127.0.0.1:50872 (error 108)
Metrics: {"server-negotiator.queue_time_us":261}
W20260928 08:49:40.807367 22934 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:49:40.796753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45938 (local address 127.0.97.61:34399)
0928 08:49:40.797110 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:40.797116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:40.804727 (+ 7611us) server_negotiation.cc:407] Connection header received
0928 08:49:40.804958 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:40.804963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:40.805032 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:40.805103 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:40.807161 (+ 2058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45938: BlockingRecv error: recv got EOF from 127.0.0.1:45938 (error 108)
Metrics: {"server-negotiator.queue_time_us":240}
W20260928 08:49:40.809788 23006 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:49:40.804867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37206 (local address 127.0.97.60:46481)
0928 08:49:40.805049 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:40.805054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:40.808211 (+ 3157us) server_negotiation.cc:407] Connection header received
0928 08:49:40.808509 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:40.808515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:40.808597 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:40.808666 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:40.809533 (+ 867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37206: BlockingRecv error: recv got EOF from 127.0.0.1:37206 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 08:49:40.826025 22934 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:49:40.817792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45948 (local address 127.0.97.61:34399)
0928 08:49:40.817960 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:40.817966 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:40.824418 (+ 6452us) server_negotiation.cc:407] Connection header received
0928 08:49:40.824672 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:40.824678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:40.824808 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:40.824939 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:40.825757 (+ 818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45948: BlockingRecv error: recv got EOF from 127.0.0.1:45948 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 08:49:40.839568 22934 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:49:40.837572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45958 (local address 127.0.97.61:34399)
0928 08:49:40.837807 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:40.837815 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:40.838308 (+ 493us) server_negotiation.cc:407] Connection header received
0928 08:49:40.838516 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:40.838522 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:40.838602 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:40.838721 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:40.839369 (+ 648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45958: BlockingRecv error: recv got EOF from 127.0.0.1:45958 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260928 08:49:40.847083 22934 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:49:40.844950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45966 (local address 127.0.97.61:34399)
0928 08:49:40.845132 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:40.845155 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:40.845664 (+ 509us) server_negotiation.cc:407] Connection header received
0928 08:49:40.845895 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:40.845904 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:40.846011 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:40.846120 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:40.846846 (+ 726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45966: BlockingRecv error: recv got EOF from 127.0.0.1:45966 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 08:49:40.855985 22934 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:49:40.853917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45982 (local address 127.0.97.61:34399)
0928 08:49:40.854193 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:40.854201 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:40.854731 (+ 530us) server_negotiation.cc:407] Connection header received
0928 08:49:40.854931 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:40.854936 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:40.855011 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:40.855112 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:40.855753 (+ 641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45982: BlockingRecv error: recv got EOF from 127.0.0.1:45982 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
W20260928 08:49:40.868979 22934 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:49:40.866866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45990 (local address 127.0.97.61:34399)
0928 08:49:40.867054 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:40.867063 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:40.867551 (+ 488us) server_negotiation.cc:407] Connection header received
0928 08:49:40.867757 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:40.867762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:40.867888 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:40.867999 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:40.868745 (+ 746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45990: BlockingRecv error: recv got EOF from 127.0.0.1:45990 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 08:49:40.894292 22934 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:49:40.892080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46000 (local address 127.0.97.61:34399)
0928 08:49:40.892344 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:40.892354 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:40.892858 (+ 504us) server_negotiation.cc:407] Connection header received
0928 08:49:40.893085 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:40.893097 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:40.893183 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:40.893296 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:40.894042 (+ 746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46000: BlockingRecv error: recv got EOF from 127.0.0.1:46000 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260928 08:49:40.932214 22934 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:49:40.929907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46012 (local address 127.0.97.61:34399)
0928 08:49:40.930145 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:40.930154 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:40.930696 (+ 542us) server_negotiation.cc:407] Connection header received
0928 08:49:40.930931 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:40.930936 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:40.931011 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:40.931134 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:40.931936 (+ 802us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46012: BlockingRecv error: recv got EOF from 127.0.0.1:46012 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260928 08:49:41.008168 22934 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:49:41.005649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46024 (local address 127.0.97.61:34399)
0928 08:49:41.005886 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:41.005893 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:41.006478 (+ 585us) server_negotiation.cc:407] Connection header received
0928 08:49:41.006708 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:41.006714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:41.006810 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:41.006920 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:41.007926 (+ 1006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46024: BlockingRecv error: recv got EOF from 127.0.0.1:46024 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260928 08:49:41.141685 22934 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:49:41.139636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46040 (local address 127.0.97.61:34399)
0928 08:49:41.139880 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:41.139887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:41.140231 (+ 344us) server_negotiation.cc:407] Connection header received
0928 08:49:41.140384 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:41.140388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:41.140464 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:41.140559 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:41.141415 (+ 856us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46040: BlockingRecv error: recv got EOF from 127.0.0.1:46040 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260928 08:49:41.407299 22934 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:49:41.405469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46042 (local address 127.0.97.61:34399)
0928 08:49:41.405754 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:41.405758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:41.405977 (+ 219us) server_negotiation.cc:407] Connection header received
0928 08:49:41.406189 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:41.406193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:41.406277 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:41.406376 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:41.407126 (+ 750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46042: BlockingRecv error: recv got EOF from 127.0.0.1:46042 (error 108)
Metrics: {"server-negotiator.queue_time_us":167}
I20260928 08:49:41.436884 23021 catalog_manager.cc:1259] Loaded cluster ID: 560e479e0d914960887f98c95a4de2a2
I20260928 08:49:41.436986 23021 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1e806cc37e1a4ca38c37aa570874bb9d: loading cluster ID for follower catalog manager: success
I20260928 08:49:41.439479 23021 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1e806cc37e1a4ca38c37aa570874bb9d: acquiring CA information for follower catalog manager: success
I20260928 08:49:41.441296 23021 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1e806cc37e1a4ca38c37aa570874bb9d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 08:49:41.739095 23068 catalog_manager.cc:1259] Loaded cluster ID: 560e479e0d914960887f98c95a4de2a2
I20260928 08:49:41.739213 23068 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3666d2df013c44bca63e7b7b1f142730: loading cluster ID for follower catalog manager: success
I20260928 08:49:41.741964 23068 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3666d2df013c44bca63e7b7b1f142730: acquiring CA information for follower catalog manager: success
I20260928 08:49:41.744012 23068 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3666d2df013c44bca63e7b7b1f142730: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 08:49:41.931358 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:41.923211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46054 (local address 127.0.97.61:34399)
0928 08:49:41.924653 (+ 1442us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:41.924657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:41.924675 (+ 18us) server_negotiation.cc:407] Connection header received
0928 08:49:41.924741 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:41.924745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:41.924866 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:41.924973 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:49:41.925538 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:41.926466 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:41.928628 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:41.929188 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:41.929315 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:41.929952 (+ 637us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:49:41.930559 (+ 607us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:49:41.930693 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:41.930821 (+ 128us) server_negotiation.cc:299] Negotiation successful
0928 08:49:41.930894 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1319,"thread_start_us":1179,"threads_started":1}
I20260928 08:49:41.945349 23140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:41.938467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37210 (local address 127.0.97.60:46481)
0928 08:49:41.939130 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:41.939134 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:41.939185 (+ 51us) server_negotiation.cc:407] Connection header received
0928 08:49:41.939274 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:41.939278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:41.939418 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:41.939551 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:49:41.939992 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:41.940747 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:41.942812 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:41.943391 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:41.943499 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:41.944099 (+ 600us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:49:41.944648 (+ 549us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:49:41.944771 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:41.944905 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 08:49:41.944976 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":452,"threads_started":1}
I20260928 08:49:41.960860 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:41.953306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46064 (local address 127.0.97.61:34399)
0928 08:49:41.953480 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:41.953486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:41.954073 (+ 587us) server_negotiation.cc:407] Connection header received
0928 08:49:41.954375 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:41.954381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:41.954471 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:41.954554 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:49:41.955206 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:41.956402 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:41.958583 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:41.959270 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:41.959359 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:41.959973 (+ 614us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:49:41.960094 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:49:41.960172 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:41.960440 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 08:49:41.960520 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 08:49:41.968319 23140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:41.954611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37226 (local address 127.0.97.60:46481)
0928 08:49:41.959162 (+ 4551us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:41.959171 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:41.961873 (+ 2702us) server_negotiation.cc:407] Connection header received
0928 08:49:41.961944 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:41.961949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:41.962030 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:41.962112 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:49:41.962710 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:41.963702 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:41.965755 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:41.966379 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:41.966500 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:41.967082 (+ 582us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:49:41.967234 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:49:41.967320 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:41.967961 (+ 641us) server_negotiation.cc:299] Negotiation successful
0928 08:49:41.968050 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4386}
I20260928 08:49:41.969354 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:41.953347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50884 (local address 127.0.97.62:40259)
0928 08:49:41.960141 (+ 6794us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:41.960148 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:41.960706 (+ 558us) server_negotiation.cc:407] Connection header received
0928 08:49:41.960903 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:41.960911 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:41.961176 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:41.961399 (+ 223us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:49:41.962413 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:41.963944 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:41.966613 (+ 2669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:41.967297 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:41.967500 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:41.968099 (+ 599us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:49:41.968579 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:49:41.968721 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:41.968929 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 08:49:41.969025 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6674,"thread_start_us":6577,"threads_started":1}
I20260928 08:49:41.974960 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 22787
I20260928 08:49:41.976352 22849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:49:42.090709 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 22787
I20260928 08:49:42.111474 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 22861
I20260928 08:49:42.112656 22923 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:49:42.219058 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 22861
I20260928 08:49:42.240715 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 22936
I20260928 08:49:42.241816 22998 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:49:42.348434 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 22936
2026-09-28T08:49:42Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4957 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 08:49:42 dist-test-slave-n6xs krb5kdc[23154](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 08:49:42 dist-test-slave-n6xs krb5kdc[23154](info): set up 2 sockets
Sep 28 08:49:42 dist-test-slave-n6xs krb5kdc[23154](info): commencing operation
krb5kdc: starting...
W20260928 08:49:43.448424 388 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/krb5kdc/test-user.keytab.
Sep 28 08:49:43 dist-test-slave-n6xs krb5kdc[23154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T08:49:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T08:49:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.97.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
Entry for principal kudu/127.0.97.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab.
I20260928 08:49:43.723409 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33025
--webserver_interface=127.0.97.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33025,127.0.97.61:42199,127.0.97.60:41199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 08:49:44.067644 23167 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 08:49:44.068044 23167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:49:44.068120 23167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:49:44.068221 23167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:49:44.075645 23167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:49:44.075771 23167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:49:44.075914 23167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:49:44.075976 23167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:49:44.076062 23167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:49:44.086520 23167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33025,127.0.97.61:42199,127.0.97.60:41199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/krb5kdc/kudu_127.0.97.62.keytab
--principal=kudu/127.0.97.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.62:33025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.97.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:49:44.087868 23167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:49:44.089162 23167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:49:44.101743 23172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:44.101789 23175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:44.101814 23173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:49:44.102630 23167 server_base.cc:1034] running on GCE node
I20260928 08:49:44.103482 23167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:49:44.105026 23167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:49:44.106282 23167 hybrid_clock.cc:648] HybridClock initialized: now 1790585384106243 us; error 55 us; skew 500 ppm
I20260928 08:49:44.106627 23167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:49:44 dist-test-slave-n6xs krb5kdc[23154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:49:44.109992 23167 init.cc:377] Logged in from keytab as kudu/127.0.97.62@KRBTEST.COM (short username kudu)
I20260928 08:49:44.111579 23167 webserver.cc:460] Webserver started at http://127.0.97.62:46863/ using document root <none> and password file <none>
I20260928 08:49:44.112380 23167 fs_manager.cc:359] Metadata directory not provided
I20260928 08:49:44.112517 23167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:49:44.112850 23167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:49:44.116001 23167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-0/data/instance:
uuid: "0c312908aa6f4a86b7e4890df05370cf"
format_stamp: "Formatted at 2026-09-28 08:49:44 on dist-test-slave-n6xs"
server_key: "1a8fc4d9ef49707bb86cd10db1e75cd1"
server_key_iv: "ffd5f008fa6653d5525006398167ce24"
server_key_version: "encryptionkey@0"
I20260928 08:49:44.116853 23167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-0/wal/instance:
uuid: "0c312908aa6f4a86b7e4890df05370cf"
format_stamp: "Formatted at 2026-09-28 08:49:44 on dist-test-slave-n6xs"
server_key: "1a8fc4d9ef49707bb86cd10db1e75cd1"
server_key_iv: "ffd5f008fa6653d5525006398167ce24"
server_key_version: "encryptionkey@0"
I20260928 08:49:44.122390 23167 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 08:49:44.126097 23182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:49:44.127660 23167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 08:49:44.127942 23167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-0/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-0/wal
uuid: "0c312908aa6f4a86b7e4890df05370cf"
format_stamp: "Formatted at 2026-09-28 08:49:44 on dist-test-slave-n6xs"
server_key: "1a8fc4d9ef49707bb86cd10db1e75cd1"
server_key_iv: "ffd5f008fa6653d5525006398167ce24"
server_key_version: "encryptionkey@0"
I20260928 08:49:44.128201 23167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:49:44.154492 23167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:49:44.158737 23167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:49:44.159058 23167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:49:44.177888 23167 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.62:33025
I20260928 08:49:44.177910 23233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.62:33025 every 8 connection(s)
I20260928 08:49:44.179319 23167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-0/data/info.pb
I20260928 08:49:44.181263 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 23167
I20260928 08:49:44.181460 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-0/wal/instance
I20260928 08:49:44.182021 388 external_mini_cluster.cc:1428] Setting key 30a5eef3c5635a519246fb279bcd76fb
I20260928 08:49:44.184214 23234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:49:44.194413 23234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33025 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.97.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.61@KRBTEST.COM" created.
Sep 28 08:49:44 dist-test-slave-n6xs krb5kdc[23154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:49:44.222209 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:44.197963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42950 (local address 127.0.97.62:33025)
0928 08:49:44.198905 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:44.198925 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:44.199409 (+ 484us) server_negotiation.cc:407] Connection header received
0928 08:49:44.201592 (+ 2183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:44.201597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:44.201851 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:44.202074 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:44.203404 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:44.204701 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:44.206221 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:44.206788 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:44.206991 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:44.213658 (+ 6667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:44.213674 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:44.213695 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:44.213748 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:44.217860 (+ 4112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:44.218603 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:44.218614 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:44.218620 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:44.218823 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:44.219255 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:44.219262 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:44.219268 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:44.220065 (+ 797us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:44.220275 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:44.220816 (+ 541us) server_negotiation.cc:299] Negotiation successful
0928 08:49:44.221242 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":104,"server-negotiator.queue_time_us":682,"thread_start_us":455,"threads_started":1}
I20260928 08:49:44.222257 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:44.197963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33025 (local address 127.0.0.1:42950)
0928 08:49:44.199242 (+ 1279us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:44.199277 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 08:49:44.201196 (+ 1919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:49:44.202096 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:49:44.202102 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:49:44.202847 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:49:44.203185 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:44.203202 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:44.204858 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:44.204863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:44.206079 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:44.206092 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:44.207043 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:44.207049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:44.207227 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:44.207969 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:49:44.208001 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:49:44.213350 (+ 5349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:49:44.218041 (+ 4691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:44.218052 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:44.218063 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:44.218458 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:44.218938 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:44.218942 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:44.218944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:44.219141 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:44.220214 (+ 1073us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:49:44.220220 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:49:44.220421 (+ 201us) client_negotiation.cc:769] Sending connection context
0928 08:49:44.220642 (+ 221us) client_negotiation.cc:240] Negotiation successful
0928 08:49:44.221042 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1018,"thread_start_us":790,"threads_started":1}
I20260928 08:49:44.225524 23234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42199 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:49:44.226585 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:44.226189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42199 (local address 127.0.0.1:37974)
0928 08:49:44.226363 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:44.226462 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 08:49:44.226925 23186 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42199: connect: Connection refused (error 111)
I20260928 08:49:44.228478 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:44.228159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42199 (local address 127.0.0.1:37984)
0928 08:49:44.228289 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:44.228361 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 08:49:44.228775 23234 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42199: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42199: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/krb5kdc/kudu_127.0.97.61.keytab.
Entry for principal kudu/127.0.97.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/krb5kdc/kudu_127.0.97.61.keytab.
I20260928 08:49:44.248039 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:42199
--webserver_interface=127.0.97.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33025,127.0.97.61:42199,127.0.97.60:41199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:49:44.278169 23234 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42199 } attempt: 1
I20260928 08:49:44.279452 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:44.278919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42199 (local address 127.0.0.1:37994)
0928 08:49:44.279194 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:44.279293 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 08:49:44.280597 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:44.280199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42199 (local address 127.0.0.1:38002)
0928 08:49:44.280357 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:44.280450 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 08:49:44.280791 23234 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42199: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42199: connect: Connection refused (error 111)
I20260928 08:49:44.349044 23234 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42199 } attempt: 2
I20260928 08:49:44.350344 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:44.349852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42199 (local address 127.0.0.1:38018)
0928 08:49:44.350104 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:44.350205 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260928 08:49:44.351361 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:44.351002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42199 (local address 127.0.0.1:38034)
0928 08:49:44.351146 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:44.351222 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 08:49:44.351531 23234 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42199: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42199: connect: Connection refused (error 111)
I20260928 08:49:44.442816 23234 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42199 } attempt: 3
I20260928 08:49:44.444204 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:44.443630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42199 (local address 127.0.0.1:38036)
0928 08:49:44.443905 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:44.444009 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 08:49:44.445353 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:44.444943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42199 (local address 127.0.0.1:38048)
0928 08:49:44.445134 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:44.445209 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 08:49:44.445544 23234 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42199: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42199: connect: Connection refused (error 111)
W20260928 08:49:44.580536 23241 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 08:49:44.580965 23241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:49:44.581096 23241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:49:44.581252 23241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 08:49:44.588830 23241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:49:44.588964 23241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
I20260928 08:49:44.588814 23234 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42199 } attempt: 4
W20260928 08:49:44.589100 23241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:49:44.589193 23241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:49:44.589308 23241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:49:44.590301 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:44.589666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42199 (local address 127.0.0.1:38060)
0928 08:49:44.589988 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:44.590140 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260928 08:49:44.591419 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:44.591022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42199 (local address 127.0.0.1:38066)
0928 08:49:44.591178 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:44.591275 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 08:49:44.591711 23234 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.61:42199: Network error: Client connection negotiation failed: client connection to 127.0.97.61:42199: connect: Connection refused (error 111)
I20260928 08:49:44.600118 23241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33025,127.0.97.61:42199,127.0.97.60:41199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/krb5kdc/kudu_127.0.97.61.keytab
--principal=kudu/127.0.97.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.61:42199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.97.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:49:44.601428 23241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:49:44.602809 23241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:49:44.615116 23249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:44.615204 23246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:44.615025 23247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:49:44.616293 23241 server_base.cc:1034] running on GCE node
I20260928 08:49:44.617148 23241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:49:44.618637 23241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:49:44.619936 23241 hybrid_clock.cc:648] HybridClock initialized: now 1790585384619864 us; error 47 us; skew 500 ppm
I20260928 08:49:44.620296 23241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:49:44 dist-test-slave-n6xs krb5kdc[23154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:49:44.623539 23241 init.cc:377] Logged in from keytab as kudu/127.0.97.61@KRBTEST.COM (short username kudu)
I20260928 08:49:44.625633 23241 webserver.cc:460] Webserver started at http://127.0.97.61:43223/ using document root <none> and password file <none>
I20260928 08:49:44.626328 23241 fs_manager.cc:359] Metadata directory not provided
I20260928 08:49:44.626415 23241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:49:44.626684 23241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:49:44.629762 23241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-1/data/instance:
uuid: "b235bf52563a4e3f8d6c17d2436e6ea2"
format_stamp: "Formatted at 2026-09-28 08:49:44 on dist-test-slave-n6xs"
server_key: "83771c7b9603f3fda8ae055eb38943df"
server_key_iv: "45fd9b30ddf133d092d8b5fcc5bafaf4"
server_key_version: "encryptionkey@0"
I20260928 08:49:44.630497 23241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-1/wal/instance:
uuid: "b235bf52563a4e3f8d6c17d2436e6ea2"
format_stamp: "Formatted at 2026-09-28 08:49:44 on dist-test-slave-n6xs"
server_key: "83771c7b9603f3fda8ae055eb38943df"
server_key_iv: "45fd9b30ddf133d092d8b5fcc5bafaf4"
server_key_version: "encryptionkey@0"
I20260928 08:49:44.635901 23241 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 08:49:44.639428 23256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:49:44.640898 23241 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 08:49:44.641080 23241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-1/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-1/wal
uuid: "b235bf52563a4e3f8d6c17d2436e6ea2"
format_stamp: "Formatted at 2026-09-28 08:49:44 on dist-test-slave-n6xs"
server_key: "83771c7b9603f3fda8ae055eb38943df"
server_key_iv: "45fd9b30ddf133d092d8b5fcc5bafaf4"
server_key_version: "encryptionkey@0"
I20260928 08:49:44.641321 23241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:49:44.655431 23241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:49:44.659929 23241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:49:44.660216 23241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:49:44.679013 23241 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.61:42199
I20260928 08:49:44.679035 23307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.61:42199 every 8 connection(s)
I20260928 08:49:44.680389 23241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-1/data/info.pb
I20260928 08:49:44.684150 23308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:49:44.684252 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 23241
I20260928 08:49:44.684473 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-1/wal/instance
I20260928 08:49:44.684850 388 external_mini_cluster.cc:1428] Setting key a95d3651bc29d9d782842f7499a369f5
I20260928 08:49:44.690949 23308 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33025 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:49:44 dist-test-slave-n6xs krb5kdc[23154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.97.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.97.60@KRBTEST.COM" created.
I20260928 08:49:44.713367 23237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:44.693698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42952 (local address 127.0.97.62:33025)
0928 08:49:44.693936 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:44.693944 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:44.695152 (+ 1208us) server_negotiation.cc:407] Connection header received
0928 08:49:44.697275 (+ 2123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:44.697283 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:44.697381 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:44.697502 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:44.698977 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:44.700180 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:44.701734 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:44.702487 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:44.702639 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:44.706255 (+ 3616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:44.706261 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:44.706266 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:44.706312 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:44.709504 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:44.710302 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:44.710338 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:44.710340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:44.710470 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:44.710997 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:44.711001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:44.711003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:44.711438 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:44.711643 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:44.712584 (+ 941us) server_negotiation.cc:299] Negotiation successful
0928 08:49:44.712861 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 08:49:44.714069 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:44.694165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33025 (local address 127.0.0.1:42952)
0928 08:49:44.694967 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:44.695017 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 08:49:44.697000 (+ 1983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:49:44.697695 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:49:44.697702 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:49:44.698279 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:49:44.698801 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:44.698817 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:44.700325 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:44.700328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:44.701578 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:44.701590 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:44.702699 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:44.702703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:44.702898 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:44.703633 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:49:44.703659 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:49:44.705996 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:49:44.709648 (+ 3652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:44.709655 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:44.709674 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:44.710114 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:44.710620 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:44.710624 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:44.710627 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:44.710833 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:44.711618 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:49:44.711626 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:49:44.712117 (+ 491us) client_negotiation.cc:769] Sending connection context
0928 08:49:44.712452 (+ 335us) client_negotiation.cc:240] Negotiation successful
0928 08:49:44.712877 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":360,"threads_started":1}
I20260928 08:49:44.716025 23308 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42199 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:49:44 dist-test-slave-n6xs krb5kdc[23154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 08:49:44.732846 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:44.717126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38080 (local address 127.0.97.61:42199)
0928 08:49:44.717789 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:44.717795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:44.717823 (+ 28us) server_negotiation.cc:407] Connection header received
0928 08:49:44.717869 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:44.717874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:44.718058 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:44.718283 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:44.719109 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:44.720327 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:44.721555 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:44.722147 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:44.722295 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:44.726160 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:44.726181 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:44.726207 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:44.726256 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:44.729428 (+ 3172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:44.730369 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:44.730376 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:44.730378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:44.730504 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:44.731005 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:44.731010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:44.731013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:44.731444 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:44.731620 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:44.732210 (+ 590us) server_negotiation.cc:299] Negotiation successful
0928 08:49:44.732481 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":510,"thread_start_us":359,"threads_started":1}
I20260928 08:49:44.732878 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:44.716813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42199 (local address 127.0.0.1:38080)
0928 08:49:44.717088 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:44.717108 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:49:44.717201 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:49:44.718272 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:49:44.718276 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:49:44.718766 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:49:44.718904 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:44.718914 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:44.720497 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:44.720505 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:44.721396 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:44.721421 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:44.722302 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:44.722308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:44.722498 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:44.723153 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:49:44.723172 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:49:44.725809 (+ 2637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:49:44.729590 (+ 3781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:44.729599 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:44.729606 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:44.730178 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:44.730627 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:44.730634 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:44.730641 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:44.730829 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:44.731576 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:49:44.731582 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:49:44.731780 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 08:49:44.731974 (+ 194us) client_negotiation.cc:240] Negotiation successful
0928 08:49:44.732370 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260928 08:49:44.735499 23308 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41199 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:49:44.736881 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:44.736288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41199 (local address 127.0.0.1:36200)
0928 08:49:44.736515 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:44.736663 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260928 08:49:44.737243 23259 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41199: connect: Connection refused (error 111)
I20260928 08:49:44.738900 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:44.738569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41199 (local address 127.0.0.1:36210)
0928 08:49:44.738712 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:44.738793 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 08:49:44.739194 23308 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41199: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41199: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/krb5kdc/kudu_127.0.97.60.keytab.
Entry for principal kudu/127.0.97.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/krb5kdc/kudu_127.0.97.60.keytab.
I20260928 08:49:44.748080 388 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
/tmp/dist-test-taskUfDo5M/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41199
--webserver_interface=127.0.97.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.97.20:46175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.97.62:33025,127.0.97.61:42199,127.0.97.60:41199
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 08:49:44.788556 23308 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41199 } attempt: 1
I20260928 08:49:44.789848 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:44.789322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41199 (local address 127.0.0.1:36220)
0928 08:49:44.789547 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:44.789657 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 08:49:44.790928 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:44.790538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41199 (local address 127.0.0.1:36234)
0928 08:49:44.790695 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:44.790801 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 08:49:44.791159 23308 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41199: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41199: connect: Connection refused (error 111)
I20260928 08:49:44.859508 23308 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41199 } attempt: 2
I20260928 08:49:44.860934 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:44.860290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41199 (local address 127.0.0.1:36244)
0928 08:49:44.860560 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:44.860703 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260928 08:49:44.862073 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:44.861671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41199 (local address 127.0.0.1:36252)
0928 08:49:44.861843 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:44.861932 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 08:49:44.862353 23308 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41199: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41199: connect: Connection refused (error 111)
I20260928 08:49:44.891047 23234 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42199 } attempt: 5
Sep 28 08:49:44 dist-test-slave-n6xs krb5kdc[23154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 08:49:44.905313 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:44.891965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42199 (local address 127.0.0.1:38082)
0928 08:49:44.892183 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:44.892213 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 08:49:44.892349 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:49:44.892826 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:49:44.892832 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:49:44.893270 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:49:44.893465 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:44.893475 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:44.894529 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:44.894536 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:44.895408 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:44.895420 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:44.896253 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:44.896258 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:44.896502 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:44.897200 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:49:44.897218 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:49:44.899576 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:49:44.902724 (+ 3148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:44.902730 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:44.902734 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:44.903141 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:44.903534 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:44.903539 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:44.903543 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:44.903687 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:44.904287 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:49:44.904292 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:49:44.904457 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 08:49:44.904606 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 08:49:44.904931 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 08:49:44.905303 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:44.892255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38082 (local address 127.0.97.61:42199)
0928 08:49:44.892489 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:44.892500 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:44.892516 (+ 16us) server_negotiation.cc:407] Connection header received
0928 08:49:44.892582 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:44.892585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:44.892669 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:44.892785 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:44.893647 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:44.894356 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:44.895585 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:44.896090 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:44.896226 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:44.899797 (+ 3571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:44.899848 (+ 51us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:44.899857 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:44.899902 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:44.902556 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:44.903272 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:44.903279 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:44.903282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:44.903413 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:44.903801 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:44.903858 (+ 57us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:44.903862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:44.904161 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:44.904322 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:44.904725 (+ 403us) server_negotiation.cc:299] Negotiation successful
0928 08:49:44.904946 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 08:49:44.906563 23234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41199 } has no permanent_uuid. Determining permanent_uuid...
I20260928 08:49:44.907622 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:44.907210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41199 (local address 127.0.0.1:36254)
0928 08:49:44.907384 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:44.907472 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 08:49:44.908552 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:44.908225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41199 (local address 127.0.0.1:36266)
0928 08:49:44.908377 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:44.908453 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 08:49:44.908804 23234 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41199: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41199: connect: Connection refused (error 111)
I20260928 08:49:44.953649 23308 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41199 } attempt: 3
I20260928 08:49:44.955147 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:44.954477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41199 (local address 127.0.0.1:36276)
0928 08:49:44.954797 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:44.954929 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260928 08:49:44.956480 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:44.955931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41199 (local address 127.0.0.1:36288)
0928 08:49:44.956176 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:44.956273 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
W20260928 08:49:44.956775 23308 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41199: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41199: connect: Connection refused (error 111)
I20260928 08:49:44.960023 23234 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41199 } attempt: 1
I20260928 08:49:44.961170 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:44.960664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41199 (local address 127.0.0.1:36298)
0928 08:49:44.960875 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:44.960979 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 08:49:44.962231 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:44.961840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41199 (local address 127.0.0.1:36304)
0928 08:49:44.962010 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:44.962097 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260928 08:49:44.962518 23234 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41199: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41199: connect: Connection refused (error 111)
I20260928 08:49:45.030776 23234 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41199 } attempt: 2
I20260928 08:49:45.032254 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:45.031551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41199 (local address 127.0.0.1:36312)
0928 08:49:45.031881 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:45.032030 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 08:49:45.033532 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:45.033075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41199 (local address 127.0.0.1:36322)
0928 08:49:45.033287 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:45.033389 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 08:49:45.033775 23234 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41199: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41199: connect: Connection refused (error 111)
W20260928 08:49:45.093798 23315 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 08:49:45.094173 23315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 08:49:45.094255 23315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 08:49:45.094339 23315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 08:49:45.100033 23308 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41199 } attempt: 4
I20260928 08:49:45.101259 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:45.100696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41199 (local address 127.0.0.1:36324)
0928 08:49:45.100961 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:45.101072 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 08:49:45.102313 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:45.101936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41199 (local address 127.0.0.1:36334)
0928 08:49:45.102103 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:45.102186 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260928 08:49:45.102424 23315 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 08:49:45.102556 23315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 08:49:45.102573 23308 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41199: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41199: connect: Connection refused (error 111)
W20260928 08:49:45.102645 23315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 08:49:45.102717 23315 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 08:49:45.102815 23315 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 08:49:45.113260 23315 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.97.20:46175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.97.62:33025,127.0.97.61:42199,127.0.97.60:41199
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/krb5kdc/kudu_127.0.97.60.keytab
--principal=kudu/127.0.97.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.97.60:41199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.97.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 07:47:32 UTC on ccycloud.quasar-uulrnl.root.comops.site
build id 4676818
ASAN enabled
I20260928 08:49:45.114589 23315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 08:49:45.116000 23315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 08:49:45.127926 23323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:45.127954 23321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 08:49:45.127887 23320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 08:49:45.128822 23315 server_base.cc:1034] running on GCE node
I20260928 08:49:45.129683 23315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 08:49:45.131095 23315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 08:49:45.132308 23315 hybrid_clock.cc:648] HybridClock initialized: now 1790585385132226 us; error 52 us; skew 500 ppm
I20260928 08:49:45.132668 23315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 08:49:45 dist-test-slave-n6xs krb5kdc[23154](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 08:49:45.135991 23315 init.cc:377] Logged in from keytab as kudu/127.0.97.60@KRBTEST.COM (short username kudu)
I20260928 08:49:45.137650 23315 webserver.cc:460] Webserver started at http://127.0.97.60:40273/ using document root <none> and password file <none>
I20260928 08:49:45.138394 23315 fs_manager.cc:359] Metadata directory not provided
I20260928 08:49:45.138530 23315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 08:49:45.138855 23315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 08:49:45.140008 23234 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41199 } attempt: 3
I20260928 08:49:45.141182 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:45.140622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41199 (local address 127.0.0.1:36346)
0928 08:49:45.140880 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:45.140981 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 08:49:45.142302 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:45.141875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41199 (local address 127.0.0.1:36348)
0928 08:49:45.142080 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:45.142165 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41199: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260928 08:49:45.142536 23234 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.97.60:41199: Network error: Client connection negotiation failed: client connection to 127.0.97.60:41199: connect: Connection refused (error 111)
I20260928 08:49:45.142787 23315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-2/data/instance:
uuid: "7c75b8f39fc94fcfb3e3f1220e32809d"
format_stamp: "Formatted at 2026-09-28 08:49:45 on dist-test-slave-n6xs"
server_key: "a45000bbeea6aa0aa22f7f61c16b6b3f"
server_key_iv: "cbc9b437e77e717748628c3371770bb8"
server_key_version: "encryptionkey@0"
I20260928 08:49:45.143635 23315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-2/wal/instance:
uuid: "7c75b8f39fc94fcfb3e3f1220e32809d"
format_stamp: "Formatted at 2026-09-28 08:49:45 on dist-test-slave-n6xs"
server_key: "a45000bbeea6aa0aa22f7f61c16b6b3f"
server_key_iv: "cbc9b437e77e717748628c3371770bb8"
server_key_version: "encryptionkey@0"
I20260928 08:49:45.149197 23315 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 08:49:45.152734 23330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 08:49:45.154345 23315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 08:49:45.154495 23315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-2/data,/tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-2/wal
uuid: "7c75b8f39fc94fcfb3e3f1220e32809d"
format_stamp: "Formatted at 2026-09-28 08:49:45 on dist-test-slave-n6xs"
server_key: "a45000bbeea6aa0aa22f7f61c16b6b3f"
server_key_iv: "cbc9b437e77e717748628c3371770bb8"
server_key_version: "encryptionkey@0"
I20260928 08:49:45.154678 23315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 08:49:45.173485 23315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 08:49:45.177953 23315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 08:49:45.178273 23315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 08:49:45.198395 23315 rpc_server.cc:307] RPC server started. Bound to: 127.0.97.60:41199
I20260928 08:49:45.198426 23381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.97.60:41199 every 8 connection(s)
I20260928 08:49:45.199911 23315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-2/data/info.pb
I20260928 08:49:45.204241 23382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 08:49:45.204864 388 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu as pid 23315
I20260928 08:49:45.205045 388 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUfDo5M/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790585022442112-388-0/minicluster-data/master-2/wal/instance
I20260928 08:49:45.205494 388 external_mini_cluster.cc:1428] Setting key 8e7a2a91c48c80208805554beb414115
I20260928 08:49:45.213724 23382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33025 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:49:45 dist-test-slave-n6xs krb5kdc[23154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 28 08:49:45 dist-test-slave-n6xs krb5kdc[23154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
I20260928 08:49:45.231853 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:45.207277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36352 (local address 127.0.97.60:41199)
0928 08:49:45.208148 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:45.208160 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:45.208208 (+ 48us) server_negotiation.cc:407] Connection header received
0928 08:49:45.210436 (+ 2228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:45.210462 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:45.210906 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:45.211274 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:45.213100 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.214749 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.216556 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.217797 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.218573 (+ 776us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:45.222410 (+ 3837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:45.222420 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:45.222436 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:45.222493 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:45.226271 (+ 3778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:45.227256 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:45.227266 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:45.227276 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:45.227488 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:45.227966 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:45.227972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:45.227978 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:45.228744 (+ 766us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:45.229064 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:45.229798 (+ 734us) server_negotiation.cc:299] Negotiation successful
0928 08:49:45.230287 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":424,"threads_started":1}
I20260928 08:49:45.235054 23387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:45.216841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.62:33025 (local address 127.0.0.1:42954)
0928 08:49:45.217743 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:45.217787 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 08:49:45.217983 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:49:45.218395 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:49:45.218402 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:49:45.219142 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:49:45.219387 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:45.219395 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.220569 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.220581 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:45.221999 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:45.222014 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.223088 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.223094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:45.223378 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:45.224107 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:49:45.224124 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:49:45.227034 (+ 2910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:49:45.231228 (+ 4194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:45.231238 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:45.231243 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:45.231893 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:45.233027 (+ 1134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:45.233033 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:45.233036 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:45.233209 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:45.233785 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:49:45.233793 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:49:45.233996 (+ 203us) client_negotiation.cc:769] Sending connection context
0928 08:49:45.234234 (+ 238us) client_negotiation.cc:240] Negotiation successful
0928 08:49:45.234611 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":538,"threads_started":1}
I20260928 08:49:45.236164 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:45.216829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42954 (local address 127.0.97.62:33025)
0928 08:49:45.217450 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:45.217457 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:45.217893 (+ 436us) server_negotiation.cc:407] Connection header received
0928 08:49:45.218136 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:45.218146 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:45.218212 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:45.218325 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:45.219541 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.220417 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.222196 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.222860 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.223004 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:45.227287 (+ 4283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:45.227294 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:45.227299 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:45.227344 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:45.230942 (+ 3598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:45.232042 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:45.232048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:45.232051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:45.232899 (+ 848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:45.233353 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:45.233365 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:45.233368 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:45.233649 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:45.233827 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:45.235399 (+ 1572us) server_negotiation.cc:299] Negotiation successful
0928 08:49:45.235697 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":312,"threads_started":1}
I20260928 08:49:45.240214 23382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42199 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:49:45 dist-test-slave-n6xs krb5kdc[23154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.62@KRBTEST.COM
Sep 28 08:49:45 dist-test-slave-n6xs krb5kdc[23154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 08:49:45.252983 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:45.235146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42962 (local address 127.0.97.62:33025)
0928 08:49:45.236428 (+ 1282us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:45.236434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:45.236454 (+ 20us) server_negotiation.cc:407] Connection header received
0928 08:49:45.236508 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:45.236513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:45.236605 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:45.236718 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:45.238737 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.239781 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.241417 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.241887 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.242033 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:45.246563 (+ 4530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:45.246573 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:45.246581 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:45.246633 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:45.250115 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:45.251001 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:45.251005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:45.251008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:45.251134 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:45.251558 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:45.251562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:45.251564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:45.251905 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:45.252232 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:45.252461 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 08:49:45.252701 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1170,"thread_start_us":3616,"threads_started":1}
I20260928 08:49:45.254784 23387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:45.240908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.61:42199 (local address 127.0.0.1:38088)
0928 08:49:45.241073 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:45.241093 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 08:49:45.241199 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:49:45.241524 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:49:45.241529 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:49:45.241966 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:49:45.242117 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:45.242126 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.243095 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.243101 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:45.244081 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:45.244093 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.244836 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.244842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:45.245053 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:45.245695 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:49:45.245715 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:49:45.248096 (+ 2381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:49:45.252072 (+ 3976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:45.252077 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:45.252081 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:45.252600 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:45.253088 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:45.253096 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:45.253100 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:45.253275 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:45.253797 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:49:45.253802 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:49:45.253968 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 08:49:45.254111 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 08:49:45.254410 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 08:49:45.254830 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:45.241023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38088 (local address 127.0.97.61:42199)
0928 08:49:45.241210 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:45.241216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:45.241237 (+ 21us) server_negotiation.cc:407] Connection header received
0928 08:49:45.241311 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:45.241316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:45.241392 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:45.241501 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:45.242280 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.242943 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.244247 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.244672 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.244801 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:45.248343 (+ 3542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:45.248351 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:45.248356 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:45.248403 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:45.251918 (+ 3515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:45.252778 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:45.252784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:45.252789 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:45.252960 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:45.253393 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:45.253401 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:45.253404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:45.253679 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:45.253837 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:45.254221 (+ 384us) server_negotiation.cc:299] Negotiation successful
0928 08:49:45.254443 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 08:49:45.256366 23382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41199 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 08:49:45 dist-test-slave-n6xs krb5kdc[23154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.60@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
Sep 28 08:49:45 dist-test-slave-n6xs krb5kdc[23154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585383, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.97.61@KRBTEST.COM
I20260928 08:49:45.276003 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:45.257173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36368 (local address 127.0.97.60:41199)
0928 08:49:45.257360 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:45.257366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:45.257494 (+ 128us) server_negotiation.cc:407] Connection header received
0928 08:49:45.257586 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:45.257591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:45.257659 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:45.257765 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:45.258530 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.259252 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.260981 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.261383 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.261478 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:45.269568 (+ 8090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:45.269574 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:45.269577 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:45.269608 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:45.273000 (+ 3392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:45.273765 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:45.273771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:45.273775 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:45.273931 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:45.274358 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:45.274363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:45.274366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:45.274731 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:45.274887 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:45.275211 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 08:49:45.275664 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":203,"server-negotiator.queue_time_us":61}
I20260928 08:49:45.276003 23387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:45.257197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41199 (local address 127.0.0.1:36368)
0928 08:49:45.257414 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:45.257429 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 08:49:45.257510 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:49:45.257790 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:49:45.257794 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:49:45.258225 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:49:45.258415 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:45.258426 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.259386 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.259392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:45.260565 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:45.260580 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.261536 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.261540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:45.261702 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:45.266413 (+ 4711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:49:45.266441 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:49:45.269335 (+ 2894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:49:45.273203 (+ 3868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:45.273208 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:45.273211 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:45.273623 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:45.274082 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:45.274086 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:45.274088 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:45.274237 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:45.274845 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:49:45.274850 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:49:45.274987 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 08:49:45.275128 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 08:49:45.275520 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":25}
I20260928 08:49:45.277915 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:45.254601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38100 (local address 127.0.97.61:42199)
0928 08:49:45.255123 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:45.255128 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:45.255143 (+ 15us) server_negotiation.cc:407] Connection header received
0928 08:49:45.255200 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:45.255204 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:45.255276 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:45.255370 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:45.256899 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.258001 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.259473 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.260026 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.260121 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:45.270986 (+ 10865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:45.270994 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:45.271001 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:45.271053 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:45.274423 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:45.275320 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:45.275328 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:45.275332 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:45.275497 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:45.276024 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:45.276030 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:45.276034 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:45.276488 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:45.276800 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:45.277225 (+ 425us) server_negotiation.cc:299] Negotiation successful
0928 08:49:45.277462 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":1337,"threads_started":1}
I20260928 08:49:45.281437 23382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c75b8f39fc94fcfb3e3f1220e32809d: Bootstrap starting.
I20260928 08:49:45.283216 388 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 08:49:45.285084 23382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c75b8f39fc94fcfb3e3f1220e32809d: Neither blocks nor log segments found. Creating new log.
I20260928 08:49:45.286527 23382 log.cc:824] T 00000000000000000000000000000000 P 7c75b8f39fc94fcfb3e3f1220e32809d: Log is configured to *not* fsync() on all Append() calls
I20260928 08:49:45.285852 23209 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0c312908aa6f4a86b7e4890df05370cf"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42962
I20260928 08:49:45.290040 23382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c75b8f39fc94fcfb3e3f1220e32809d: No bootstrap required, opened a new log
I20260928 08:49:45.301344 23382 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c75b8f39fc94fcfb3e3f1220e32809d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c312908aa6f4a86b7e4890df05370cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33025 } } peers { permanent_uuid: "b235bf52563a4e3f8d6c17d2436e6ea2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42199 } } peers { permanent_uuid: "7c75b8f39fc94fcfb3e3f1220e32809d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41199 } }
I20260928 08:49:45.301829 23382 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c75b8f39fc94fcfb3e3f1220e32809d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c75b8f39fc94fcfb3e3f1220e32809d, State: Initialized, Role: FOLLOWER
I20260928 08:49:45.302364 23382 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c75b8f39fc94fcfb3e3f1220e32809d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c312908aa6f4a86b7e4890df05370cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33025 } } peers { permanent_uuid: "b235bf52563a4e3f8d6c17d2436e6ea2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42199 } } peers { permanent_uuid: "7c75b8f39fc94fcfb3e3f1220e32809d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41199 } }
I20260928 08:49:45.303354 23391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c75b8f39fc94fcfb3e3f1220e32809d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c312908aa6f4a86b7e4890df05370cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33025 } } peers { permanent_uuid: "b235bf52563a4e3f8d6c17d2436e6ea2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42199 } } peers { permanent_uuid: "7c75b8f39fc94fcfb3e3f1220e32809d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41199 } } }
I20260928 08:49:45.303730 23391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c75b8f39fc94fcfb3e3f1220e32809d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:49:45.304811 23382 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c75b8f39fc94fcfb3e3f1220e32809d [sys.catalog]: configured and running, proceeding with master startup.
W20260928 08:49:45.315300 23402 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7c75b8f39fc94fcfb3e3f1220e32809d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 08:49:45.315466 23402 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 08:49:45.319784 23234 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41199 } attempt: 4
I20260928 08:49:45.313000 23315 master_runner.cc:186] Error getting master registration for 127.0.97.62:33025: OK, instance_id { permanent_uuid: "0c312908aa6f4a86b7e4890df05370cf" instance_seqno: 1790585384172649 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 08:49:45.324831 23315 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 28 08:49:45 dist-test-slave-n6xs krb5kdc[23154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.62@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260928 08:49:45.338074 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:45.320605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41199 (local address 127.0.0.1:36380)
0928 08:49:45.321019 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:45.321043 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 08:49:45.321192 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:49:45.322200 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:49:45.322205 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:49:45.322729 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:49:45.322958 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:45.322970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.324730 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.324736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:45.325742 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:45.325755 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.326875 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.326884 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:45.327111 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:45.327871 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:49:45.327896 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:49:45.330768 (+ 2872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:49:45.335114 (+ 4346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:45.335120 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:45.335124 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:45.335673 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:45.336139 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:45.336143 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:45.336147 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:45.336330 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:45.336992 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:49:45.336996 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:49:45.337180 (+ 184us) client_negotiation.cc:769] Sending connection context
0928 08:49:45.337327 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 08:49:45.337659 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270}
I20260928 08:49:45.338086 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:45.321425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36380 (local address 127.0.97.60:41199)
0928 08:49:45.321660 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:45.321667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:45.321697 (+ 30us) server_negotiation.cc:407] Connection header received
0928 08:49:45.321776 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:45.321781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:45.321872 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:45.321991 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:45.323177 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.324519 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.325912 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.326666 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.326807 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:45.331000 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:45.331009 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:45.331014 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:45.331053 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:45.334934 (+ 3881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:45.335841 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:45.335848 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:45.335852 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:45.336010 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:45.336447 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:45.336452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:45.336455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:45.336858 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:45.337021 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:45.337451 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 08:49:45.337704 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 08:49:45.344484 23234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c312908aa6f4a86b7e4890df05370cf: Bootstrap starting.
I20260928 08:49:45.348790 23234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c312908aa6f4a86b7e4890df05370cf: Neither blocks nor log segments found. Creating new log.
I20260928 08:49:45.350225 23234 log.cc:824] T 00000000000000000000000000000000 P 0c312908aa6f4a86b7e4890df05370cf: Log is configured to *not* fsync() on all Append() calls
I20260928 08:49:45.354044 23234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c312908aa6f4a86b7e4890df05370cf: No bootstrap required, opened a new log
I20260928 08:49:45.362329 23234 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c312908aa6f4a86b7e4890df05370cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c312908aa6f4a86b7e4890df05370cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33025 } } peers { permanent_uuid: "b235bf52563a4e3f8d6c17d2436e6ea2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42199 } } peers { permanent_uuid: "7c75b8f39fc94fcfb3e3f1220e32809d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41199 } }
I20260928 08:49:45.362824 23234 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c312908aa6f4a86b7e4890df05370cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c312908aa6f4a86b7e4890df05370cf, State: Initialized, Role: FOLLOWER
I20260928 08:49:45.363397 23234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c312908aa6f4a86b7e4890df05370cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c312908aa6f4a86b7e4890df05370cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33025 } } peers { permanent_uuid: "b235bf52563a4e3f8d6c17d2436e6ea2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42199 } } peers { permanent_uuid: "7c75b8f39fc94fcfb3e3f1220e32809d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41199 } }
I20260928 08:49:45.364481 23405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c312908aa6f4a86b7e4890df05370cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c312908aa6f4a86b7e4890df05370cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33025 } } peers { permanent_uuid: "b235bf52563a4e3f8d6c17d2436e6ea2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42199 } } peers { permanent_uuid: "7c75b8f39fc94fcfb3e3f1220e32809d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41199 } } }
I20260928 08:49:45.364898 23405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c312908aa6f4a86b7e4890df05370cf [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:49:45.365947 23234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c312908aa6f4a86b7e4890df05370cf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:49:45.366147 23209 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c312908aa6f4a86b7e4890df05370cf [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 08:49:45.366366 23209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c312908aa6f4a86b7e4890df05370cf [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:49:45.369092 23209 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c312908aa6f4a86b7e4890df05370cf [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c312908aa6f4a86b7e4890df05370cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33025 } } peers { permanent_uuid: "b235bf52563a4e3f8d6c17d2436e6ea2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42199 } } peers { permanent_uuid: "7c75b8f39fc94fcfb3e3f1220e32809d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41199 } }
I20260928 08:49:45.371304 23357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0c312908aa6f4a86b7e4890df05370cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7c75b8f39fc94fcfb3e3f1220e32809d"
I20260928 08:49:45.371770 23357 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c75b8f39fc94fcfb3e3f1220e32809d [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:49:45.374280 23357 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7c75b8f39fc94fcfb3e3f1220e32809d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c312908aa6f4a86b7e4890df05370cf in term 1.
I20260928 08:49:45.374569 23209 leader_election.cc:288] T 00000000000000000000000000000000 P 0c312908aa6f4a86b7e4890df05370cf [CANDIDATE]: Term 1 election: Requested vote from peers b235bf52563a4e3f8d6c17d2436e6ea2 (127.0.97.61:42199), 7c75b8f39fc94fcfb3e3f1220e32809d (127.0.97.60:41199)
I20260928 08:49:45.375001 23185 leader_election.cc:302] T 00000000000000000000000000000000 P 0c312908aa6f4a86b7e4890df05370cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c312908aa6f4a86b7e4890df05370cf, 7c75b8f39fc94fcfb3e3f1220e32809d; no voters:
I20260928 08:49:45.375543 23405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c312908aa6f4a86b7e4890df05370cf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 08:49:45.388557 23405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c312908aa6f4a86b7e4890df05370cf [term 1 LEADER]: Becoming Leader. State: Replica: 0c312908aa6f4a86b7e4890df05370cf, State: Running, Role: LEADER
I20260928 08:49:45.389467 23405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c312908aa6f4a86b7e4890df05370cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c312908aa6f4a86b7e4890df05370cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33025 } } peers { permanent_uuid: "b235bf52563a4e3f8d6c17d2436e6ea2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42199 } } peers { permanent_uuid: "7c75b8f39fc94fcfb3e3f1220e32809d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41199 } }
I20260928 08:49:45.401782 23308 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41199 } attempt: 5
I20260928 08:49:45.406616 23409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c312908aa6f4a86b7e4890df05370cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c312908aa6f4a86b7e4890df05370cf. Latest consensus state: current_term: 1 leader_uuid: "0c312908aa6f4a86b7e4890df05370cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c312908aa6f4a86b7e4890df05370cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33025 } } peers { permanent_uuid: "b235bf52563a4e3f8d6c17d2436e6ea2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42199 } } peers { permanent_uuid: "7c75b8f39fc94fcfb3e3f1220e32809d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41199 } } }
I20260928 08:49:45.407277 23409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c312908aa6f4a86b7e4890df05370cf [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:45.421154 23422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260928 08:49:45.421738 23188 tablet.cc:2369] T 00000000000000000000000000000000 P 0c312908aa6f4a86b7e4890df05370cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 08:49:45.430670 23336 tablet.cc:2369] T 00000000000000000000000000000000 P 7c75b8f39fc94fcfb3e3f1220e32809d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 08:49:45 dist-test-slave-n6xs krb5kdc[23154](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790585384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.97.61@KRBTEST.COM for kudu/127.0.97.60@KRBTEST.COM
I20260928 08:49:45.451973 23422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 08:49:45.470310 23385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:45.398223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36384 (local address 127.0.97.60:41199)
0928 08:49:45.398477 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:45.398491 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:45.422894 (+ 24403us) server_negotiation.cc:407] Connection header received
0928 08:49:45.423020 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:45.423029 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:45.423162 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:45.423359 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:45.433910 (+ 10551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.435268 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.442580 (+ 7312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.443351 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.443482 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:45.454415 (+ 10933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:45.454425 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:45.454433 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:45.454490 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:45.462039 (+ 7549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:45.467195 (+ 5156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:45.467208 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:45.467214 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:45.467410 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:45.468007 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:45.468015 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:45.468022 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:45.468475 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:45.468792 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:45.469457 (+ 665us) server_negotiation.cc:299] Negotiation successful
0928 08:49:45.469780 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 08:49:45.470315 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:45.399677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42966 (local address 127.0.97.62:33025)
0928 08:49:45.399945 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:45.399954 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:45.427530 (+ 27576us) server_negotiation.cc:407] Connection header received
0928 08:49:45.428085 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:45.428096 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:45.428232 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:45.428371 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:45.433998 (+ 5627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.435561 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.442280 (+ 6719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.443025 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.443135 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:45.454466 (+ 11331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:45.454473 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:45.454477 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:45.454572 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:45.463453 (+ 8881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:45.466100 (+ 2647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:45.466108 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:45.466112 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:45.466440 (+ 328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:45.467160 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:45.467167 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:45.467171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:45.467843 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:45.468263 (+ 420us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:45.469149 (+ 886us) server_negotiation.cc:299] Negotiation successful
0928 08:49:45.469545 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 08:49:45.476400 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:45.427687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38104 (local address 127.0.97.61:42199)
0928 08:49:45.427947 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:45.427957 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:45.439639 (+ 11682us) server_negotiation.cc:407] Connection header received
0928 08:49:45.440264 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:45.440271 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:45.440357 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:45.440429 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:45.451136 (+ 10707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.452263 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.455379 (+ 3116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.456077 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.456184 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:45.461059 (+ 4875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:45.461067 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:45.461072 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:45.461120 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:45.464787 (+ 3667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:45.473120 (+ 8333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:45.473127 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:45.473130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:45.473309 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:45.473980 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:45.473988 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:45.473992 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:45.474346 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:45.474587 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:45.475500 (+ 913us) server_negotiation.cc:299] Negotiation successful
0928 08:49:45.475743 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 08:49:45.477274 23311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:45.402555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.97.60:41199 (local address 127.0.0.1:36392)
0928 08:49:45.403020 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 08:49:45.403046 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 08:49:45.403192 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 08:49:45.429007 (+ 25815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 08:49:45.429016 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 08:49:45.429758 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0928 08:49:45.430044 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:45.430062 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.438695 (+ 8633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.438707 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:45.440732 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 08:49:45.440754 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.444048 (+ 3294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.444059 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 08:49:45.444352 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:45.445314 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 08:49:45.445348 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 08:49:45.449175 (+ 3827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 08:49:45.470872 (+ 21697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:45.470882 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:45.470889 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:45.471590 (+ 701us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:45.474798 (+ 3208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 08:49:45.474806 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 08:49:45.474812 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 08:49:45.475020 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 08:49:45.475926 (+ 906us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 08:49:45.475934 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 08:49:45.476148 (+ 214us) client_negotiation.cc:769] Sending connection context
0928 08:49:45.476361 (+ 213us) client_negotiation.cc:240] Negotiation successful
0928 08:49:45.476767 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316}
I20260928 08:49:45.478830 23421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:45.403450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36392 (local address 127.0.97.60:41199)
0928 08:49:45.428423 (+ 24973us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:45.428432 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:45.428463 (+ 31us) server_negotiation.cc:407] Connection header received
0928 08:49:45.428536 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:45.428543 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:45.428642 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:45.430251 (+ 1609us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:45.431987 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.433633 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.441107 (+ 7474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.442087 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.449733 (+ 7646us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:45.450036 (+ 303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 08:49:45.450046 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 08:49:45.450053 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 08:49:45.450092 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 08:49:45.466994 (+ 16902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:45.474368 (+ 7374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:45.474376 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:45.474381 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:45.474597 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 08:49:45.475187 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 08:49:45.475194 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 08:49:45.475200 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 08:49:45.475709 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 08:49:45.477775 (+ 2066us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:45.477930 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 08:49:45.478231 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24784,"thread_start_us":25310,"threads_started":1}
I20260928 08:49:45.488771 23357 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7c75b8f39fc94fcfb3e3f1220e32809d [term 1 FOLLOWER]: Refusing update from remote peer 0c312908aa6f4a86b7e4890df05370cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:49:45.486788 23308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b235bf52563a4e3f8d6c17d2436e6ea2: Bootstrap starting.
I20260928 08:49:45.491775 23405 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c312908aa6f4a86b7e4890df05370cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c75b8f39fc94fcfb3e3f1220e32809d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:49:45.501085 23308 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b235bf52563a4e3f8d6c17d2436e6ea2: Neither blocks nor log segments found. Creating new log.
I20260928 08:49:45.503561 23308 log.cc:824] T 00000000000000000000000000000000 P b235bf52563a4e3f8d6c17d2436e6ea2: Log is configured to *not* fsync() on all Append() calls
I20260928 08:49:45.493868 23167 master_runner.cc:186] Error getting master registration for 127.0.97.61:42199: OK, instance_id { permanent_uuid: "b235bf52563a4e3f8d6c17d2436e6ea2" instance_seqno: 1790585384673527 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 08:49:45.515172 23167 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 08:49:45.524796 23308 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b235bf52563a4e3f8d6c17d2436e6ea2: No bootstrap required, opened a new log
I20260928 08:49:45.530052 23391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c75b8f39fc94fcfb3e3f1220e32809d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c312908aa6f4a86b7e4890df05370cf. Latest consensus state: current_term: 1 leader_uuid: "0c312908aa6f4a86b7e4890df05370cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c312908aa6f4a86b7e4890df05370cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33025 } } peers { permanent_uuid: "b235bf52563a4e3f8d6c17d2436e6ea2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42199 } } peers { permanent_uuid: "7c75b8f39fc94fcfb3e3f1220e32809d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41199 } } }
I20260928 08:49:45.530383 23391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c75b8f39fc94fcfb3e3f1220e32809d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:49:45.537376 23430 mvcc.cc:205] Tried to move back new op lower bound from 7334237738937606144 to 7334237738563063808. Current Snapshot: MvccSnapshot[applied={T|T < 7334237738937606144}]
I20260928 08:49:45.544464 23391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c75b8f39fc94fcfb3e3f1220e32809d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0c312908aa6f4a86b7e4890df05370cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c312908aa6f4a86b7e4890df05370cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33025 } } peers { permanent_uuid: "b235bf52563a4e3f8d6c17d2436e6ea2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42199 } } peers { permanent_uuid: "7c75b8f39fc94fcfb3e3f1220e32809d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41199 } } }
I20260928 08:49:45.544740 23391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c75b8f39fc94fcfb3e3f1220e32809d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:49:45.546334 23409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c312908aa6f4a86b7e4890df05370cf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0c312908aa6f4a86b7e4890df05370cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c312908aa6f4a86b7e4890df05370cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33025 } } peers { permanent_uuid: "b235bf52563a4e3f8d6c17d2436e6ea2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42199 } } peers { permanent_uuid: "7c75b8f39fc94fcfb3e3f1220e32809d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41199 } } }
I20260928 08:49:45.546607 23409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c312908aa6f4a86b7e4890df05370cf [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:45.547855 23422 catalog_manager.cc:1347] Generated new cluster ID: ff7dec0078624740a80ab6bc21bf58e8
I20260928 08:49:45.547977 23422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 08:49:45.571748 23308 raft_consensus.cc:348] T 00000000000000000000000000000000 P b235bf52563a4e3f8d6c17d2436e6ea2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c312908aa6f4a86b7e4890df05370cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33025 } } peers { permanent_uuid: "b235bf52563a4e3f8d6c17d2436e6ea2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42199 } } peers { permanent_uuid: "7c75b8f39fc94fcfb3e3f1220e32809d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41199 } }
I20260928 08:49:45.572409 23308 raft_consensus.cc:729] T 00000000000000000000000000000000 P b235bf52563a4e3f8d6c17d2436e6ea2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b235bf52563a4e3f8d6c17d2436e6ea2, State: Initialized, Role: FOLLOWER
I20260928 08:49:45.573129 23308 consensus_queue.cc:260] T 00000000000000000000000000000000 P b235bf52563a4e3f8d6c17d2436e6ea2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c312908aa6f4a86b7e4890df05370cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33025 } } peers { permanent_uuid: "b235bf52563a4e3f8d6c17d2436e6ea2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42199 } } peers { permanent_uuid: "7c75b8f39fc94fcfb3e3f1220e32809d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41199 } }
I20260928 08:49:45.585214 23283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0c312908aa6f4a86b7e4890df05370cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b235bf52563a4e3f8d6c17d2436e6ea2"
I20260928 08:49:45.585541 23282 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b235bf52563a4e3f8d6c17d2436e6ea2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 08:49:45.589154 23282 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b235bf52563a4e3f8d6c17d2436e6ea2 [term 1 FOLLOWER]: Refusing update from remote peer 0c312908aa6f4a86b7e4890df05370cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 08:49:45.590116 23409 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0c312908aa6f4a86b7e4890df05370cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b235bf52563a4e3f8d6c17d2436e6ea2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 08:49:45.591161 23308 sys_catalog.cc:564] T 00000000000000000000000000000000 P b235bf52563a4e3f8d6c17d2436e6ea2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 08:49:45.594614 23283 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b235bf52563a4e3f8d6c17d2436e6ea2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c312908aa6f4a86b7e4890df05370cf in term 1.
I20260928 08:49:45.584487 23436 sys_catalog.cc:455] T 00000000000000000000000000000000 P b235bf52563a4e3f8d6c17d2436e6ea2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c312908aa6f4a86b7e4890df05370cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33025 } } peers { permanent_uuid: "b235bf52563a4e3f8d6c17d2436e6ea2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42199 } } peers { permanent_uuid: "7c75b8f39fc94fcfb3e3f1220e32809d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41199 } } }
I20260928 08:49:45.601033 23436 sys_catalog.cc:458] T 00000000000000000000000000000000 P b235bf52563a4e3f8d6c17d2436e6ea2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:49:45.609462 23422 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 08:49:45.612341 23422 catalog_manager.cc:1504] Loading token signing keys...
I20260928 08:49:45.649431 23438 sys_catalog.cc:455] T 00000000000000000000000000000000 P b235bf52563a4e3f8d6c17d2436e6ea2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0c312908aa6f4a86b7e4890df05370cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c312908aa6f4a86b7e4890df05370cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33025 } } peers { permanent_uuid: "b235bf52563a4e3f8d6c17d2436e6ea2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42199 } } peers { permanent_uuid: "7c75b8f39fc94fcfb3e3f1220e32809d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41199 } } }
I20260928 08:49:45.649902 23438 sys_catalog.cc:458] T 00000000000000000000000000000000 P b235bf52563a4e3f8d6c17d2436e6ea2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:49:45.650307 23436 sys_catalog.cc:455] T 00000000000000000000000000000000 P b235bf52563a4e3f8d6c17d2436e6ea2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c312908aa6f4a86b7e4890df05370cf. Latest consensus state: current_term: 1 leader_uuid: "0c312908aa6f4a86b7e4890df05370cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c312908aa6f4a86b7e4890df05370cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33025 } } peers { permanent_uuid: "b235bf52563a4e3f8d6c17d2436e6ea2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42199 } } peers { permanent_uuid: "7c75b8f39fc94fcfb3e3f1220e32809d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41199 } } }
I20260928 08:49:45.650563 23436 sys_catalog.cc:458] T 00000000000000000000000000000000 P b235bf52563a4e3f8d6c17d2436e6ea2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 08:49:45.652495 23435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c312908aa6f4a86b7e4890df05370cf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0c312908aa6f4a86b7e4890df05370cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c312908aa6f4a86b7e4890df05370cf" member_type: VOTER last_known_addr { host: "127.0.97.62" port: 33025 } } peers { permanent_uuid: "b235bf52563a4e3f8d6c17d2436e6ea2" member_type: VOTER last_known_addr { host: "127.0.97.61" port: 42199 } } peers { permanent_uuid: "7c75b8f39fc94fcfb3e3f1220e32809d" member_type: VOTER last_known_addr { host: "127.0.97.60" port: 41199 } } }
I20260928 08:49:45.653059 23435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c312908aa6f4a86b7e4890df05370cf [sys.catalog]: This master's current role is: LEADER
I20260928 08:49:45.678220 23422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c312908aa6f4a86b7e4890df05370cf: Generated new TSK 0
I20260928 08:49:45.679283 23422 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 08:49:45.695472 23451 catalog_manager.cc:1259] Loaded cluster ID: ff7dec0078624740a80ab6bc21bf58e8
I20260928 08:49:45.695665 23451 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b235bf52563a4e3f8d6c17d2436e6ea2: loading cluster ID for follower catalog manager: success
I20260928 08:49:45.699462 23451 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b235bf52563a4e3f8d6c17d2436e6ea2: acquiring CA information for follower catalog manager: success
I20260928 08:49:45.701958 23451 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b235bf52563a4e3f8d6c17d2436e6ea2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 08:49:45.806793 23421 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:49:45.797198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36408 (local address 127.0.97.60:41199)
0928 08:49:45.797429 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:45.797436 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:45.803075 (+ 5639us) server_negotiation.cc:407] Connection header received
0928 08:49:45.805080 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:45.805087 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:45.805228 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:45.805333 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:45.806542 (+ 1209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36408: BlockingRecv error: recv got EOF from 127.0.0.1:36408 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260928 08:49:45.811702 23386 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:49:45.797980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42970 (local address 127.0.97.62:33025)
0928 08:49:45.798187 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:45.798193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:45.808971 (+ 10778us) server_negotiation.cc:407] Connection header received
0928 08:49:45.810221 (+ 1250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:45.810227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:45.810309 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:45.810387 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:45.811433 (+ 1046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42970: BlockingRecv error: recv got EOF from 127.0.0.1:42970 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260928 08:49:45.813685 23312 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:49:45.803596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38120 (local address 127.0.97.61:42199)
0928 08:49:45.803865 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:45.803871 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:45.809595 (+ 5724us) server_negotiation.cc:407] Connection header received
0928 08:49:45.812181 (+ 2586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:45.812192 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:45.812285 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:45.812359 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:45.813440 (+ 1081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38120: BlockingRecv error: recv got EOF from 127.0.0.1:38120 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 08:49:45.862393 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:45.846447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38130 (local address 127.0.97.61:42199)
0928 08:49:45.846674 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:45.846680 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:45.853946 (+ 7266us) server_negotiation.cc:407] Connection header received
0928 08:49:45.854484 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:45.854491 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:45.854726 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:45.854875 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:49:45.855613 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.856779 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.859280 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.859992 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.860098 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:45.860863 (+ 765us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:49:45.861568 (+ 705us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:49:45.861681 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:45.861925 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 08:49:45.862026 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260928 08:49:45.922689 23421 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:49:45.900055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36412 (local address 127.0.97.60:41199)
0928 08:49:45.900356 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:45.900366 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:45.920547 (+ 20181us) server_negotiation.cc:407] Connection header received
0928 08:49:45.920874 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:45.920882 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:45.921011 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:45.921128 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:45.922369 (+ 1241us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36412: BlockingRecv error: recv got EOF from 127.0.0.1:36412 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
I20260928 08:49:45.980502 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:45.964438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38134 (local address 127.0.97.61:42199)
0928 08:49:45.964652 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:45.964659 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:45.971656 (+ 6997us) server_negotiation.cc:407] Connection header received
0928 08:49:45.972290 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:45.972297 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:45.972425 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:45.972511 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:49:45.973304 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.974551 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.976719 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:45.978596 (+ 1877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:45.978723 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:45.979468 (+ 745us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:49:45.979663 (+ 195us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:49:45.979748 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:45.980045 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 08:49:45.980152 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 08:49:46.024946 23421 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:49:46.014828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36428 (local address 127.0.97.60:41199)
0928 08:49:46.015097 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:46.015104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:46.022731 (+ 7627us) server_negotiation.cc:407] Connection header received
0928 08:49:46.023356 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:46.023363 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:46.023467 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:46.023544 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:46.024702 (+ 1158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36428: BlockingRecv error: recv got EOF from 127.0.0.1:36428 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 08:49:46.076333 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:46.063347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38136 (local address 127.0.97.61:42199)
0928 08:49:46.063544 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:46.063550 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:46.069936 (+ 6386us) server_negotiation.cc:407] Connection header received
0928 08:49:46.070179 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:46.070187 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:46.070309 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:46.070415 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:49:46.070981 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:46.072126 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:46.074109 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:46.074773 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:46.074934 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:46.075418 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:49:46.075571 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:49:46.075675 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:46.075847 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 08:49:46.075958 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260928 08:49:46.084995 23421 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:49:46.082832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36440 (local address 127.0.97.60:41199)
0928 08:49:46.083073 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:46.083079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:46.083462 (+ 383us) server_negotiation.cc:407] Connection header received
0928 08:49:46.083649 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:46.083653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:46.083726 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:46.083855 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:46.084741 (+ 886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36440: BlockingRecv error: recv got EOF from 127.0.0.1:36440 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 08:49:46.097345 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:46.092037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38144 (local address 127.0.97.61:42199)
0928 08:49:46.092358 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:46.092364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:46.092592 (+ 228us) server_negotiation.cc:407] Connection header received
0928 08:49:46.092777 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:46.092781 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:46.092858 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:46.092994 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:49:46.093471 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:46.094277 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:46.095553 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:46.096095 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:46.096234 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:46.096614 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:49:46.096712 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:49:46.096812 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:46.096964 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 08:49:46.097042 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197}
W20260928 08:49:46.105057 23421 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:49:46.103034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36454 (local address 127.0.97.60:41199)
0928 08:49:46.103286 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:46.103291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:46.103521 (+ 230us) server_negotiation.cc:407] Connection header received
0928 08:49:46.103723 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:46.103726 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:46.103855 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:46.103948 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:46.104770 (+ 822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36454: BlockingRecv error: recv got EOF from 127.0.0.1:36454 (error 108)
Metrics: {"server-negotiator.queue_time_us":141}
I20260928 08:49:46.122665 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:46.116935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38158 (local address 127.0.97.61:42199)
0928 08:49:46.117153 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:46.117160 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:46.117390 (+ 230us) server_negotiation.cc:407] Connection header received
0928 08:49:46.117543 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:46.117554 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:46.117647 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:46.117754 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:49:46.118121 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:46.119268 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:46.120710 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:46.121352 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:46.121477 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:46.121890 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:49:46.122031 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:49:46.122132 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:46.122281 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 08:49:46.122367 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 08:49:46.131108 23421 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:49:46.128913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36464 (local address 127.0.97.60:41199)
0928 08:49:46.129189 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:46.129196 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:46.129510 (+ 314us) server_negotiation.cc:407] Connection header received
0928 08:49:46.129696 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:46.129700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:46.129793 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:46.129891 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:46.130833 (+ 942us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36464: BlockingRecv error: recv got EOF from 127.0.0.1:36464 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
I20260928 08:49:46.157051 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:46.151079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38162 (local address 127.0.97.61:42199)
0928 08:49:46.151281 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:46.151288 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:46.151737 (+ 449us) server_negotiation.cc:407] Connection header received
0928 08:49:46.151973 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:46.151980 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:46.152064 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:46.152182 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:49:46.152533 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:46.153743 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:46.155112 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:46.155755 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:46.155908 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:46.156304 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:49:46.156447 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:49:46.156544 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:46.156682 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 08:49:46.156769 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 08:49:46.164985 23421 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:49:46.163055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36474 (local address 127.0.97.60:41199)
0928 08:49:46.163250 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:46.163256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:46.163569 (+ 313us) server_negotiation.cc:407] Connection header received
0928 08:49:46.163720 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:46.163726 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:46.163845 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:46.163957 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:46.164734 (+ 777us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36474: BlockingRecv error: recv got EOF from 127.0.0.1:36474 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 08:49:46.214464 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:46.207319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38178 (local address 127.0.97.61:42199)
0928 08:49:46.207564 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:46.207571 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:46.208187 (+ 616us) server_negotiation.cc:407] Connection header received
0928 08:49:46.208402 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:46.208409 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:46.208507 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:46.208616 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:49:46.209073 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:46.210252 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:46.212268 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:46.212944 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:46.213066 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:46.213556 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:49:46.213710 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:49:46.213837 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:46.214009 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 08:49:46.214104 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 08:49:46.226979 23421 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:49:46.224812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36484 (local address 127.0.97.60:41199)
0928 08:49:46.225003 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:46.225009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:46.225473 (+ 464us) server_negotiation.cc:407] Connection header received
0928 08:49:46.225657 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:46.225662 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:46.225772 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:46.225910 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:46.226730 (+ 820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36484: BlockingRecv error: recv got EOF from 127.0.0.1:36484 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 08:49:46.300299 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:46.294504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38180 (local address 127.0.97.61:42199)
0928 08:49:46.294697 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:46.294704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:46.295137 (+ 433us) server_negotiation.cc:407] Connection header received
0928 08:49:46.295301 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:46.295307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:46.295397 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:46.295506 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:49:46.295900 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:46.297010 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:46.298399 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:46.298870 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:46.298983 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:46.299360 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:49:46.299466 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:49:46.299544 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:46.299773 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 08:49:46.299922 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260928 08:49:46.308459 23421 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 08:49:46.306334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36486 (local address 127.0.97.60:41199)
0928 08:49:46.306509 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:46.306514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:46.306832 (+ 318us) server_negotiation.cc:407] Connection header received
0928 08:49:46.307119 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:46.307125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:46.307248 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:46.307344 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 08:49:46.308197 (+ 853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36486: BlockingRecv error: recv got EOF from 127.0.0.1:36486 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 08:49:46.317441 23402 catalog_manager.cc:1259] Loaded cluster ID: ff7dec0078624740a80ab6bc21bf58e8
I20260928 08:49:46.317539 23402 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7c75b8f39fc94fcfb3e3f1220e32809d: loading cluster ID for follower catalog manager: success
I20260928 08:49:46.320152 23402 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7c75b8f39fc94fcfb3e3f1220e32809d: acquiring CA information for follower catalog manager: success
I20260928 08:49:46.321990 23402 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7c75b8f39fc94fcfb3e3f1220e32809d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 08:49:46.444821 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:46.439899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38188 (local address 127.0.97.61:42199)
0928 08:49:46.440154 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:46.440160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:46.440424 (+ 264us) server_negotiation.cc:407] Connection header received
0928 08:49:46.440553 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:46.440557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:46.440644 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:46.440727 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:49:46.441142 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:46.441852 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:46.443226 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:46.443633 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:46.443733 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:46.444101 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:49:46.444227 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:49:46.444301 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:46.444408 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 08:49:46.444470 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 08:49:46.456941 23421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:46.450972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36500 (local address 127.0.97.60:41199)
0928 08:49:46.451156 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:46.451160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:46.451441 (+ 281us) server_negotiation.cc:407] Connection header received
0928 08:49:46.451600 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:46.451604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:46.451770 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:46.451940 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:49:46.452303 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:46.453273 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:46.454636 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:46.455039 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:46.455140 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:46.455731 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:49:46.456386 (+ 655us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:49:46.456503 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:46.456636 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 08:49:46.456705 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 08:49:46.471504 23421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:46.463078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36514 (local address 127.0.97.60:41199)
0928 08:49:46.463375 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:46.463384 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:46.464434 (+ 1050us) server_negotiation.cc:407] Connection header received
0928 08:49:46.464704 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:46.464709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:46.464812 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:46.464887 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:49:46.465788 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:46.466763 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:46.469389 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:46.470063 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:46.470175 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:46.470700 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:49:46.470862 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:49:46.470950 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:46.471101 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 08:49:46.471189 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260928 08:49:46.471866 23312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:46.464531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38192 (local address 127.0.97.61:42199)
0928 08:49:46.464718 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:46.464724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:46.465277 (+ 553us) server_negotiation.cc:407] Connection header received
0928 08:49:46.465578 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:46.465586 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:46.465677 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:46.465769 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:49:46.466376 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:46.467423 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:46.469263 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:46.469934 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:46.470068 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:46.471082 (+ 1014us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:49:46.471228 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:49:46.471314 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:46.471507 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 08:49:46.471592 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 08:49:46.472697 23558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 08:49:46.462931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42980 (local address 127.0.97.62:33025)
0928 08:49:46.463828 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0928 08:49:46.463837 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 08:49:46.464500 (+ 663us) server_negotiation.cc:407] Connection header received
0928 08:49:46.464663 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 08:49:46.464673 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 08:49:46.464925 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 08:49:46.465119 (+ 194us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 08:49:46.465860 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:46.466982 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:46.470264 (+ 3282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 08:49:46.470742 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 08:49:46.470866 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 08:49:46.471424 (+ 558us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 08:49:46.471978 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 08:49:46.472085 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 08:49:46.472278 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 08:49:46.472366 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":579,"threads_started":1}
I20260928 08:49:46.477897 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 23167
I20260928 08:49:46.478854 23229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:49:46.591215 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 23167
I20260928 08:49:46.613456 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 23241
I20260928 08:49:46.614647 23303 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:49:46.724627 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 23241
I20260928 08:49:46.745530 388 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu pid 23315
I20260928 08:49:46.746704 23377 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 08:49:46.856142 388 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUfDo5M/build/asan/bin/kudu with pid 23315
2026-09-28T08:49:46Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4512 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9469 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (364377 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST